Pictures and text on the screen

Games, maps and some custom-drawn apps show no elements, only one big box in the dump. Coordinates work there, but they break when the layout moves. Two optional commands look at the screenshot instead:

• find_image and tap_image look for a small picture you cut from an earlier screenshot.
• ocr, ocr_find and ocr_tap read the text on the screen with Tesseract.

Both run on the PC that runs droidline serve: the server takes a screenshot from the phone and does the work there, so the phone needs nothing new. Use elements whenever the dump has them; they are faster and exact.

Find a picture and tap it

1. Take a screenshot at full size: droidline screenshot screen.png.
2. Open it in any image editor and cut out the button or icon you want, a little tighter than its edges. Save it as PNG.
3. Look for it:

spot = d.find_image("play-button.png")
print(spot["x"], spot["y"], spot["score"])     # center, and how close the match was

d.tap_image("play-button.png")                  # find it, then tap its center
d.tap_image("next.png", threshold=0.8, timeout=20)

const spot = await d.findImage("play-button.png");
console.log(spot.x, spot.y, spot.score);

await d.tapImage("play-button.png");
await d.tapImage("next.png", { threshold: 0.8, timeout: 20 });

var spot = await d.FindImageAsync("play-button.png");
Console.WriteLine($"{spot.X} {spot.Y} {spot.Score}");

await d.TapImageAsync("play-button.png");
await d.TapImageAsync("next.png", threshold: 0.8, timeout: 20);

droidline find_image play-button.png
droidline tap_image play-button.png
droidline tap_image next.png --threshold 0.8 --timeout 20

The SDKs also take the picture's bytes instead of a path.

Both commands try again until the picture shows up or timeout runs out (10 seconds by default), then fail with NOT_FOUND. The message gives the best score it saw, which tells you how far off it was.

What makes a match:

• Cut it from the same phone. The picture is compared pixel for pixel at the same size, so a picture from another screen size or density does not match. Keep one set of pictures per phone model.
• Cut something distinctive. A plain colored square matches in many places; the label or the icon on it matches in one.
• Adjust threshold with care. It runs from 0 to 1 and defaults to 0.9. Lower it a little when a button changes slightly, for example a glow or a counter. Too low and it taps something else.

Read the text with OCR

OCR needs Tesseract on the PC that runs the server, along with the language data you read: eng for English, kor for Korean, chi_sim for simplified Chinese. Droidline does not install it. When tesseract is not on PATH, set its location in config.toml:

[ocr]
tesseract = "C:/Program Files/Tesseract-OCR/tesseract.exe"

Then:

page = d.ocr("kor+eng")
print(page["text"])                       # every line, one per row
for line in page["lines"]:
    print(line["text"], line["bounds"])

d.ocr_find("Level 12", "eng")             # wait until it shows, return where
d.ocr_tap("Start", "eng", timeout=30)     # wait, then tap it

const page = await d.ocr("kor+eng");
console.log(page.text);
for (const line of page.lines) console.log(line.text, line.bounds);

await d.ocrFind("Level 12", "eng");
await d.ocrTap("Start", "eng", { timeout: 30 });

var page = await d.OcrAsync("kor+eng");
Console.WriteLine(page.Text);
foreach (var line in page.Lines)
    Console.WriteLine($"{line.Get<string>("text")} [{string.Join(", ", line.Get<int[]>("bounds")!)}]");

await d.OcrFindAsync("Level 12", "eng");
await d.OcrTapAsync("Start", "eng", timeout: 30);

droidline ocr kor+eng
droidline ocr_find "Level 12" eng
droidline ocr_tap Start eng --timeout 30

ocr_find and ocr_tap first look for a whole word equal to your text, then for a line that contains it. Without Tesseract they fail with OCR_UNAVAILABLE, saying what is missing.

OCR reads what the picture shows, so it is slower than elements and makes mistakes with small, stylized or low-contrast text. Each try takes a screenshot and runs Tesseract once, which takes much longer than looking up an element.