Finding elements

Most of the work in a Droidline script is telling it which element to act on. This guide shows how to look at a screen the way Droidline does, and how to choose selectors that keep working when the app updates or the phone's language changes.

Look before you write

Open the screen on the phone, then save what Droidline sees:

droidline dump screen.json
droidline screenshot screen.png

d.dump("screen.json")
d.screenshot("screen.png")

await d.dump("screen.json");
await d.screenshot("screen.png");

Open screen.json in an editor next to the screenshot and search for the words you can see on the phone. Each match is an element; the fields around it are your options:

{
  "text": "Keep me signed in",
  "id": "dev.droidline.demo:id/auto_login",
  "desc": "",
  "class": "android.widget.CheckBox",
  "bounds": [60, 860, 1020, 960],
  "clickable": true,
  "checkable": true,
  "checked": false
}

bounds is [left, top, right, bottom] in pixels. If you are not sure which element in the file is which on the screen, compare bounds with the screenshot: the element above sits from 60 to 1020 pixels across and 860 to 960 down.

Choose a selector

A selector is a field and a value. Choose in this order:

1. id, when the element has one. IDs are set by the app's developers and do not change with the phone's language. Most survive app updates. Write just the part after :id/: ("id", "auto_login").
2. text, for buttons and labels. Easy to read in your code, but it is the visible wording, so it changes with the language and sometimes with an update. Use textContains when the text includes something that varies, such as a count: ("textContains", "unread") matches "3 unread".
3. desc, for icons. Icon buttons usually show no text, but have a description for screen readers, such as "Search" or "More options".
4. class with nth, as a last resort. "The third checkbox" breaks as soon as the layout changes.

d.touch("id", "auto_login")
d.touch("text", "Log in")
d.touch("textContains", "unread")
d.touch("desc", "More options")
d.touch("class", "android.widget.CheckBox", nth=2)

await d.touch("id", "auto_login");
await d.touch("text", "Log in");
await d.touch("textContains", "unread");
await d.touch("desc", "More options");
await d.touch("class", "android.widget.CheckBox", { nth: 2 });

droidline touch id auto_login
droidline touch text "Log in"
droidline touch textContains unread
droidline touch desc "More options"
droidline touch class android.widget.CheckBox --nth 2

Text you cannot tap directly

Often the text is a label inside a bigger clickable row, and the label itself is not clickable. You do not need to find the row: touch notices that the label refuses the click and taps its nearest clickable parent instead. The reply says "via": "parent" when that happened.

If a label sits next to the control you want, such as a switch at the end of a settings row, tap the label's row and the switch usually toggles. If not, select the switch itself, often with class and nth.

Several elements match

When more than one element matches, commands act on the first one in screen order. Use nth to pick another, counting from 0, and count to see how many there are:

print(d.count("text", "Delete"))          # 3
d.touch("text", "Delete", nth=1)          # the second one

console.log(await d.count("text", "Delete"));   // 3
await d.touch("text", "Delete", { nth: 1 });    // the second one

droidline count text Delete
droidline touch text Delete --nth 1

Elements further down a list

A dump only contains what is on the screen, plus a little more for some lists. An element further down does not exist yet, so touch cannot find it. scroll_to swipes until it appears, then you act on it:

d.scroll_to("text", "Developer options")
d.touch("text", "Developer options")

d.scroll_to("text", "Airplane mode", direction="up")   # search upward

await d.scrollTo("text", "Developer options");
await d.touch("text", "Developer options");

await d.scrollTo("text", "Airplane mode", { direction: "up" });   // search upward

droidline scroll_to text "Developer options"
droidline touch text "Developer options"

It gives up after 20 swipes (max_swipes) with NOT_FOUND.

Pop-ups, keyboards and the status bar

By default a dump shows the app in front. Pop-up dialogs belong to it and show up normally. To also see system windows such as the keyboard, the status bar or a permission dialog drawn by Android itself, ask for every window:

d.dump("all.json", all_windows=True)

await d.dump("all.json", { all_windows: true });

droidline dump all.json --all_windows

For pop-ups that may or may not appear, check with a condition instead of waiting for them:

if d.exists("text", "Allow"):
    d.touch("text", "Allow")

if (await d.exists("text", "Allow")) await d.touch("text", "Allow");

[ "$(droidline exists text Allow)" = "true" ] && droidline touch text Allow

Web pages and web views

Pages in Chrome and web views inside apps usually expose their links, buttons and form fields as elements too, with the visible text in text and sometimes the field's label in desc. Treat them like any other screen: dump first. If a web page shows nothing useful in the dump, it is drawn on a canvas; use coordinates as described next.

Screens without elements

Games, video players, maps and some custom-drawn apps do not expose elements at all: the dump shows one big box. On those screens, work with coordinates and colors:

d.tap(540, 1650)                         # tap a point, in screen pixels
print(d.color(540, 1650))                # "#FF6B21": the color at that point
d.swipe(540, 1600, 540, 400, 300)        # drag from one point to another in 300 ms

await d.tap(540, 1650);
console.log(await d.color(540, 1650));
await d.swipe(540, 1600, 540, 400, 300);

droidline tap 540 1650
droidline color 540 1650
droidline swipe 540 1600 540 400 300

Take coordinates from a screenshot at full size: the pixel positions in the image are the phone's screen pixels. color lets you check that a button is there before you tap, for example waiting until a pixel turns the expected color.

When a selector does not match

touch fails with NOT_FOUND after its timeout and tells you what was on the screen instead:

NOT_FOUND: Could not find text 'Log in' within 10s. Current screen: com.example / .MainActivity

Go through these in order:

1. Is it the screen you expected? The message names the app and screen that were showing. A pop-up, an ad or a slow network may have put you somewhere else.
2. Is the text exactly right? Matching is exact, including capital letters, spaces and punctuation. Copy the value from the dump instead of typing it. Use textContains if part of it varies.
3. Is it on the screen yet? Below the fold, use scroll_to. Still loading, pass a longer timeout.
4. Is it in another window? Dump with all_windows=True.
5. Does the screen have elements at all? If the dump is one big box, use coordinates.