Finding elements

How to read a screen dump, choose the selector that keeps working, handle duplicates, lists, icons, web pages and screens without elements, and debug a selector that does not match.

Most of the work in a Droidline script is telling it which element to act on. This guide shows how to look at a screen the way Droidline does, and how to choose selectors that keep working when the app updates or the phone's language changes.

Look before you write

Open the screen on the phone, then save what Droidline sees:

droidline dump screen.json
droidline screenshot screen.png
d.dump("screen.json")
d.screenshot("screen.png")
await d.dump("screen.json");
await d.screenshot("screen.png");
await d.DumpAsync("screen.json");
await d.ScreenshotAsync("screen.png");

Open screen.json in an editor next to the screenshot and search for the words you can see on the phone. Each match is an element; the fields around it are your options:

{
  "text": "Keep me signed in",
  "id": "dev.droidline.demo:id/auto_login",
  "desc": "",
  "class": "android.widget.CheckBox",
  "bounds": [60, 860, 1020, 960],
  "clickable": true,
  "checkable": true,
  "checked": false
}

bounds is [left, top, right, bottom] in pixels. If you are not sure which element in the file is which on the screen, compare bounds with the screenshot: the element above sits from 60 to 1020 pixels across and 860 to 960 down.

If you would rather point and click, droidline inspect shows the screenshot and the tree side by side and suggests selectors with code. It is optional; the dump above is all you need.

Choose a selector

A selector is a field and a value. Choose in this order:

  1. id, when the element has one. IDs are set by the app's developers and do not change with the phone's language. Most survive app updates. Write just the part after :id/: ("id", "auto_login").
  2. text, for buttons and labels. Easy to read in your code, but it is the visible wording, so it changes with the language and sometimes with an update. Use textContains when the text includes something that varies, such as a count: ("textContains", "unread") matches "3 unread".
  3. desc, for icons. Icon buttons usually show no text, but have a description for screen readers, such as "Search" or "More options".
  4. A query, when one field matches several elements. Combine conditions, or name the element by what is next to it. See When one field is not enough.
  5. class with nth, as a last resort. "The third checkbox" breaks as soon as the layout changes.
d.touch("id", "auto_login")
d.touch("text", "Log in")
d.touch("textContains", "unread")
d.touch("desc", "More options")
d.touch("class", "android.widget.CheckBox", nth=2)
await d.touch("id", "auto_login");
await d.touch("text", "Log in");
await d.touch("textContains", "unread");
await d.touch("desc", "More options");
await d.touch("class", "android.widget.CheckBox", { nth: 2 });
await d.TouchAsync("id", "auto_login");
await d.TouchAsync("text", "Log in");
await d.TouchAsync("textContains", "unread");
await d.TouchAsync("desc", "More options");
await d.TouchAsync("class", "android.widget.CheckBox", nth: 2);
droidline touch id auto_login
droidline touch text "Log in"
droidline touch textContains unread
droidline touch desc "More options"
droidline touch class android.widget.CheckBox --nth 2

Text you cannot tap directly

Often the text is a label inside a bigger clickable row, and the label itself is not clickable. You do not need to find the row: touch notices that the label refuses the click and taps its nearest clickable parent instead. The reply says "via": "parent" when that happened.

If a label sits next to the control you want, such as a switch at the end of a settings row, tap the label's row and the switch usually toggles. If not, select the switch itself with a query such as {"class": "android.widget.Switch", "row": {"text": "Wi-Fi"}}.

Several elements match

When more than one element matches, commands act on the first one in screen order. Use nth to pick another, counting from 0, and count to see how many there are:

print(d.count("text", "Delete"))          # 3
d.touch("text", "Delete", nth=1)          # the second one
console.log(await d.count("text", "Delete"));   // 3
await d.touch("text", "Delete", { nth: 1 });    // the second one
Console.WriteLine(await d.CountAsync("text", "Delete"));   // 3
await d.TouchAsync("text", "Delete", nth: 1);              // the second one
droidline count text Delete
droidline touch text Delete --nth 1

When one field is not enough

A selector can also be an object of conditions, called a query. Pass it where the field name goes and leave the value out. Every condition must hold on the same element:

d.touch({"class": "android.widget.Button", "text": "OK"})
d.checked({"class": "android.widget.Switch", "row": {"text": "Wi-Fi"}})
d.input({"editable": True, "below": {"text": "Email"}}, "knife")
await d.touch({ class: "android.widget.Button", text: "OK" });
await d.checked({ class: "android.widget.Switch", row: { text: "Wi-Fi" } });
await d.input({ editable: true, below: { text: "Email" } }, "knife");
await d.TouchAsync(new { @class = "android.widget.Button", text = "OK" });
await d.CheckedAsync(new { @class = "android.widget.Switch", row = new { text = "Wi-Fi" } });
await d.InputAsync(new { editable = true, below = new { text = "Email" } }, "knife");
droidline touch '{"class":"android.widget.Button","text":"OK"}'
droidline checked '{"class":"android.widget.Switch","row":{"text":"Wi-Fi"}}'
droidline input '{"editable":true,"below":{"text":"Email"}}' knife

A query can hold:

  • Fields: text, textContains, textMatches, id, desc, descContains, descMatches, class and package, matched the same way as the selectors above.
  • Flags: clickable, checkable, checked, enabled, editable, selected, scrollable and the others in a dump, set to true or false. One of them, visible, is false for list items kept just past the edge of the screen. Matching ignores it until you add "visible": true, which is worth doing before you tap an element's coordinates yourself.
  • Relations, each holding another query:
    • has: one of its descendants matches. A clickable row that holds the text "Wi-Fi" is {"clickable": true, "has": {"text": "Wi-Fi"}}.
    • inside: one of its ancestors matches.
    • row: something in the same list row matches. This is the way to reach a switch or a checkbox by the label next to it.
    • below, above, left_of, right_of: it sits on that side of the first match. Matches come back nearest first, so nth=0 is the closest.

Every command that takes a selector takes a query: touch, exists, wait, get_text, count, scroll_to, which and the rest. The command reference lists every key.

Regular expressions

textMatches and descMatches take a regular expression that must match the whole text, not a part of it:

d.touch("textMatches", r"Delete \d+ items?")
d.wait("descMatches", "Download(ing|ed).*")
await d.touch("textMatches", "Delete \\d+ items?");
await d.wait("descMatches", "Download(ing|ed).*");
await d.TouchAsync("textMatches", "Delete \\d+ items?");
await d.WaitAsync("descMatches", "Download(ing|ed).*");
droidline touch textMatches 'Delete \d+ items?'
droidline wait descMatches 'Download(ing|ed).*'

Use textContains when a plain part of the text is enough; it is easier to read.

Keep an element and use it

find waits for an element like touch does, then gives it back as an object instead of tapping it. find_all returns every match on the screen right away. Read their fields, act on them, or look inside them:

for row in d.find_all("id", "message_row"):
    sender = row.find("id", "sender").text
    if sender == "Mom":
        row.click()
        break

field = d.find({"editable": True})
print(field.bounds, field["enabled"])
field.input("hello")
for (const row of await d.findAll("id", "message_row")) {
  const sender = (await row.find("id", "sender")).text;
  if (sender === "Mom") {
    await row.click();
    break;
  }
}

const field = await d.find({ editable: true });
console.log(field.bounds, field.node.enabled);
await field.input("hello");
foreach (var row in await d.FindAllAsync("id", "message_row"))
{
    var sender = (await row.FindAsync("id", "sender")).Text;
    if (sender == "Mom")
    {
        await row.ClickAsync();
        break;
    }
}

var field = await d.FindAsync(new { editable = true });
Console.WriteLine($"{field.Bounds} {field["enabled"]}");
await field.InputAsync("hello");

An element is a snapshot of what was on the screen. Its actions find it again by its exact position and class, so once the screen scrolls or changes they fail with NOT_FOUND. Call refresh() for its current fields, or find it again. In the CLI, droidline find and droidline find_all print the same fields as JSON.

Elements further down a list

A dump only contains what is on the screen, plus a little more for some lists. Those extra items have "visible": false. An element further down does not exist yet, so touch cannot find it. scroll_to swipes until it is on the screen, then you act on it:

d.scroll_to("text", "Developer options")
d.touch("text", "Developer options")

d.scroll_to("text", "Airplane mode", direction="up")   # search upward
await d.scrollTo("text", "Developer options");
await d.touch("text", "Developer options");

await d.scrollTo("text", "Airplane mode", { direction: "up" });   // search upward
await d.ScrollToAsync("text", "Developer options");
await d.TouchAsync("text", "Developer options");

await d.ScrollToAsync("text", "Airplane mode", direction: "up");   // search upward
droidline scroll_to text "Developer options"
droidline touch text "Developer options"

It gives up after 20 swipes (max_swipes) with NOT_FOUND.

Pop-ups, keyboards and the status bar

By default a dump shows the app in front. Pop-up dialogs belong to it and show up normally. To also see system windows such as the keyboard, the status bar or a permission dialog drawn by Android itself, ask for every window:

d.dump("all.json", all_windows=True)
await d.dump("all.json", { all_windows: true });
await d.DumpAsync("all.json", allWindows: true);
droidline dump all.json --all_windows

For pop-ups that may or may not appear, check with a condition instead of waiting for them:

if d.exists("text", "Allow"):
    d.touch("text", "Allow")
if (await d.exists("text", "Allow")) await d.touch("text", "Allow");
if (await d.ExistsAsync("text", "Allow"))
    await d.TouchAsync("text", "Allow");
[ "$(droidline exists text Allow)" = "true" ] && droidline touch text Allow

Web pages and web views

Pages in Chrome and web views inside apps usually expose their links, buttons and form fields as elements too, with the visible text in text and sometimes the field's label in desc. Treat them like any other screen: dump first. If a web page shows nothing useful in the dump, it is drawn on a canvas; use coordinates as described next.

Screens without elements

Games, video players, maps and some custom-drawn apps do not expose elements at all: the dump shows one big box. On those screens, work with coordinates and colors:

d.tap(540, 1650)                         # tap a point, in screen pixels
print(d.color(540, 1650))                # "#FF6B21": the color at that point
d.swipe(540, 1600, 540, 400, 300)        # drag from one point to another in 300 ms
await d.tap(540, 1650);
console.log(await d.color(540, 1650));
await d.swipe(540, 1600, 540, 400, 300);
await d.TapAsync(540, 1650);
Console.WriteLine(await d.ColorAsync(540, 1650));
await d.SwipeAsync(540, 1600, 540, 400, 300);
droidline tap 540 1650
droidline color 540 1650
droidline swipe 540 1600 540 400 300

Take coordinates from a screenshot at full size: the pixel positions in the image are the phone's screen pixels. color lets you check that a button is there before you tap, for example waiting until a pixel turns the expected color.

When a selector does not match

touch fails with NOT_FOUND after its timeout and tells you what was on the screen instead:

NOT_FOUND: Could not find text 'Log in' within 10s. Current screen: com.example / .MainActivity

Go through these in order:

  1. Is it the screen you expected? The message names the app and screen that were showing. A pop-up, an ad or a slow network may have put you somewhere else.
  2. Is the text exactly right? Matching is exact, including capital letters, spaces and punctuation. Copy the value from the dump instead of typing it. Use textContains if part of it varies.
  3. Is it on the screen yet? Below the fold, use scroll_to. Still loading, pass a longer timeout.
  4. Is it in another window? Dump with all_windows=True.
  5. Does the screen have elements at all? If the dump is one big box, use coordinates, or find a picture or a text on the screenshot.