Parser data from the Internet on Arduino

Topics / Advice

Hello everyone, in this homemade I will show how you can make a cheap, simple, but at the same time efficient platform-based data parser from the Internet Arduino with Wifi module. With it, you can easily find out almost any information from the network in seconds. This can be the number of subscribers on youtube, the number of incoming messages, whether they published a homemade product, etc.

Here is a simple diagram to assemble.

Before starting to read the article, I would recommend watching a video showing the possibilities of homemade work and its practical application.

We will need:
-The platform is based on the ESP family, in my cases, and this is WeMos D1 R2

-LCD display for information output, I have a popular LCD 1602 with I2C module
-wire and button (although you may not need it)
And
-USB cable (for downloading firmware)
-WiFI access point
- optional
- own site and domain (for handling complex queries of popular sites)

Before proceeding to the next steps, make sure that the environment through which you will download the sketch is supported by the ESP microcontroller family. To configure the Arduino IDE, go to habr and follow the installation instructions. Link:

https://habr.com/en/post/371853/

Detailed manufacturing description:
Conventionally, the stages can be divided into 2 parts.
1 Assembly of the "iron" part
2 Downloading the program and setting it up

The assembly of the first stage ends very quickly. We collect it from the image above and that's it.
Display module
VCC - 5V
GND - GND
SCL - D15
SDA - D14

Buttun - GND to D7

All these settings can be changed in the sketch itself, if you have a different platform or version of the microcontroller.

The second stage is a bit more complicated, open the Arduino IDE, select the appropriate board and indicate the port on which it is located. In my cases, the board: WeMos D1 R1 and port 3

Parser data from the Internet on Arduino

All other settings can be left by default.
The sketch itself:

Show / Hide Text

/ *
Sketch for parsing information from sites, and further display

Sketch written by MEGAVOLT channel https://www.youtube.com/MEGAVOLT

Version v0.1, if there are any errors, please write a comment under the video

* /
// SETTINGS
const char * ssid = "RT-2.4GHz_WiFi_AD08"; // wifi point name
const char * password = "NqkDATh3"; // password wifi points
const String A = "http://jsonplaceholder.typicode.com/users/1,name,User 1:; http: //jsonplaceholder.typicode.com/users/2,name,User 2 :;";
String DataArr [9] [3] = {};
const bool anotherLine = true; // output information on different lines
const byte interruptPin = D7; // button pin
const int delay_ = 5000;
// *************
#include
#include
#include
#include
#include
#include

int countElem, cur;
unsigned long currentMillis;

DynamicJsonDocument doc (1024);
LiquidCrystal_I2C lcd (0x27.16.2);

class Parse {

public:

static void Execute () {
HTTPClient http; // Declare an object of class HTTPClient

http.begin (DataArr [cur] [0]); // Specify request destination
int httpCode = http.GET (); // Send the request

if (httpCode> 0) {// Check the returning code
deserializeJson (doc, http.getString ());
String echo = DataArr [cur] [2];
String payload = http.getString (); // Get the request response payload
Serial.println (echo);
String DataJ = doc [DataArr [cur] [1]]; // Print the response payload
Parse :: printPaint (DataJ, echo + ":");
} else {Serial.print ("Error"); Serial.println (httpCode);}

http.end (); // Close connection
}

static void printPaint (String Print = "", String Param = "") {
lcd.clear ();
lcd.setCursor (0, 0);
if (anotherLine) {
lcd.print (Param);
lcd.setCursor (0, 1); lcd.print (Print);
} else lcd.print (Param + Print);


}

static void explodeStr () {
String sourse = A;
char delemiter = ';';
countElem = -1;

while (sourse.indexOf (';')! = -1) {
countElem ++;
String tmpText = sourse.substring (0, sourse.indexOf (';'));
sourse.replace (tmpText + ";", "");
String urlGet = tmpText.substring (0, tmpText.indexOf (',')); tmpText.replace (urlGet + ",", "");
String Data = tmpText.substring (0, tmpText.indexOf (',')); tmpText.replace (Data + ",", "");
String echoText = tmpText.substring (0, tmpText.indexOf (',')); tmpText.replace (echoText + ",", "");
Serial.print (urlGet); Serial.print (""); Serial.print (Data); Serial.print (""); Serial.print (echoText);
Serial.println ("-----------");
DataArr [countElem] [0] = urlGet;
DataArr [countElem] [1] = Data;
DataArr [countElem] [2] = echoText;
}

}
};


void setup () {

Serial.begin (115200);
WiFi.begin (ssid, password);
lcd.init ();
// turn on LCD backlight
lcd.backlight ();
pinMode (interruptPin, INPUT_PULLUP);

currentMillis = millis ();
Parse :: explodeStr ();
Parse :: printPaint ("", "Connect,");
int i = 0;
while (WiFi.status ()! = WL_CONNECTED) {
Serial.print ("Connecting ..");
lcd.setCursor (i, 1); lcd.print (".");
i ++;
delay (1000);

}
Serial.println ("Connect successful");

}

void B () {
if (WiFi.status () == WL_CONNECTED) {// Check WiFi connection status
Parse :: Execute ();
}
}

void loop () {

if (! digitalRead (interruptPin)) {
cur ++; if (cur> countElem) cur = 0;
Parse :: printPaint ("wait", "Please,");
currentMillis = millis ();
B ();

delay (300);
}

if (millis () - currentMillis> = delay_)
{
currentMillis = millis ();
B ();

}



}

Please note that you must have 2 additional libraries installed. These are ArduinoJson.h and LiquidCrystal_I2C.h !

Then we change the important variables in the sketch settings, these are SSID (name of the Wifi point) and password (password of the Wifi point)

After which, you can download the firmware.
If the display has been connected correctly, the following information will be displayed on it.

If this does not happen, check the following setting.

Please enter a valid I2C address c. In any case, it is better the first time after downloading the sketch to open the Serial port, and make sure that the connection has been successfully established.

If this does not happen, double-check the data of the WiFi point.

If everything has successfully booted and connected, then on the displays you should see the following:

You can also click on the button and check if the data is updated (if you connected it).
Everything, at this stage 95% of the work is done, it remains only to indicate in the configuration the data that needs to be parsed. They are all in constant "A"

const String A = "http://jsonplaceholder.typicode.com/users/1,name,User 1:; http: //jsonplaceholder.typicode.com/users/2,name,User 2 :;";

So, the syntax for writing is simple. All sources are divided into groups separated by semicolons (;).
Each group consists of 3 main parameters, this
- URL to which the request will go
- The value to be retrieved.
- Additional text displayed.
Thus

http://jsonplaceholder.typicode.com/users/1,name,User 1:

==
URL = http://jsonplaceholder.typicode.com/users/1
KEY = name
text = "User 1:"

If you follow the link now

http://jsonplaceholder.typicode.com/users/1

Then you can see the following information

Where, in fact, you can see where the "name" value came from.

Thus, you can already extract information from simple sites. But this method is not suitable for working with "serious" sites that work with SSL certificates, which check requests and display any captcha if they don’t like something, and besides, the data output is too complicated to process by forces arduino. " As I have not tried to "get around" all this, but there are things that either work very unstable or do not work at all. Constant certificate updates do not help much. Therefore, as I believe, the most reasonable solution is to create your own website, "inside" of which all these requests will be executed, processed and displayed by arduino itself in a "simplified" form. How to create a site there are many articles / videos / forums, etc., at the present time this is not a problem. Suppose you have one, then upload the following file to it. Call it what you like, well, let's say "pars.php".

Parser data from the Internet on Arduino

Suitable for topic

Related topics

Add a comment

DIY categories