Summary Never-ending Learning of User Interfaces arxiv.org
11,564 words - PDF document - View PDF document
One Line
The Never-ending UI Learner is an automated system that learns about user interfaces by installing and exploring real apps, with a focus on challenging elements like tappability and dragging, using a coordinator-worker architecture.
Slides
Slide Presentation (12 slides)
Key Points
- The Never-ending UI Learner is an app crawler that automatically installs real apps from a mobile app store and crawls them to infer semantic properties of user interfaces (UIs).
- The system uses a coordinator-worker architecture to download and crawl publicly available apps, with the coordinator server maintaining a list of app IDs to crawl and tracking successful and unsuccessful crawls.
- The crawler contains screen-level and element-level models to understand UI content and generate semantic representations.
- The performance of the crawler was evaluated over five crawl epochs, with three variations of the crawler tested using different crawling strategies.
- The tappability heuristic involves taking screenshots before and after a tap to identify visual changes, with multiple screenshots used to reduce false positives.
- The model trained on human-annotated data performed poorly when predicting the tappability of elements, suggesting that the heuristic-annotated data is of higher quality.
- Screen similarity models have been used in various software engineering applications, such as mobile app usage videos, automated software testing, and automated storyboard generation.
- The authors conducted experiments to study the effects of retraining frequency on the performance of UI understanding models, suggesting that less frequent updates may be beneficial.
Summaries
29 word summary
The Never-ending UI Learner automatically learns about user interfaces by installing and crawling real apps, focusing on difficult aspects such as tappability and dragging. It uses a coordinator-worker architecture.
40 word summary
The Never-ending UI Learner is a system that automatically installs and crawls real apps to learn about user interfaces (UIs). It focuses on learning difficult aspects of UI semantics such as tappability and dragging. The system uses a coordinator-worker architecture
584 word summary
The Never-ending UI Learner is an app crawler that automatically installs real apps from a mobile app store and crawls them to infer semantic properties of user interfaces (UIs). It interacts with UI elements to discover new training examples and continually updates machine learning
The Never-ending UI Learner is a system that can continuously learn from user interfaces (UIs) by crawling the app ecosystem and collecting data from all available apps. It focuses on learning difficult aspects of UI semantics such as tappability, dragg
The Never-ending UI Learner is a system that uses a coordinator-worker architecture to automatically download and crawl publicly available apps. The coordinator server maintains a list of app IDs to crawl and tracks successful and unsuccessful crawls. Crawler workers download and install target
The document discusses a crawler system that uploads raw data and processed output to a coordinator server. The number of crawler workers varied between 40-100. The crawler contains screen-level and element-level models to understand UI content and generate semantic representations. The screen
The text discusses the performance of a crawler in analyzing user interface (UI) elements on mobile devices. Three variations of the crawler were tested, each with a different crawling strategy. The performance of the crawler was evaluated over five crawl epochs. The experiments collected
The tappability heuristic involves taking screenshots of a screen before and after a tap to identify visual changes. Multiple screenshots are used to reduce false positives. The accuracy of the heuristic was validated against human-labeled interaction videos, with an overall accuracy of
The model trained on human-annotated data performed poorly when predicting the tappability of elements in a crawled dataset. The disagreement between human-annotated and crawler-generated labels suggests that the heuristic-annotated data is of higher quality. Dragging
The draggability model computes loss on elements affected by dragging, ignoring those that did not move. It uses embeddings from the element detector and a single-layer transformer to provide additional context. Annotators were shown pre-drag, post-drag,
Screen similarity models have been used in various software engineering applications, such as mobile app usage videos, automated software testing, and automated storyboard generation. The authors of the document mined additional examples of same-screen pairs to augment the training data for their screen similarity model
The baseline model improved the initial model to a final F1 score of 0.659. The crawler-augmented dataset achieved a final F1 score of 0.663 while the baseline's final F1 score was 0.659.
The authors of the paper conducted experiments over the span of one month, with updates every 1-2 days, to study the effects of retraining frequency on the performance of UI understanding models. They suggest that less frequent updates, such as monthly updates
The summary is not provided.
This summary provides a list of references to various research papers related to user interfaces and mobile app design. The papers cover topics such as sketch-based user interface retrieval, overcoming catastrophic forgetting in neural networks, mining Android user interfaces at scale, topic modeling of mobile
This text is a list of references and citations from various sources related to the topic of user interfaces and machine learning. The sources include academic papers, books, and conference proceedings. These references provide additional resources for further research and exploration in the field of user
The document contains citations for various research papers related to user interfaces and deep learning. It includes information on modeling mobile interface tappability, efficient model scaling for convolutional neural networks, data augmentation methods for improving sentence scoring tasks, a reinforcement learning platform for