ESP8266 support for websvf

[udif]

I just left PR #1 to start this effort.
My ultimate target (I would love to hear your feedback) is to be able to reuse old ESP8266-01 for JTAG programming via WiFi. I don't want to store any bitstream file on the SPIFFS, just send it to the FPGA on-the-fly. If I need to slow it a bit to get it to work reliably with WiFi, so be it.
Is this feasible with the single core ESP8266?
As I want to be able to reuse tons of old modules lying around, I really hope to be able to squeeze it into 512KB SPI memories.
Currently websvf compiles to 373,160 bytes , hopefully leaving room for a small SPIFFS just for the minimal read-only web files.
As I haven't used ESP8266 in the few years, I need to do some catch-up on SPIFFS and other ESP8266 stuff, so I haven't ran this code yet other than getting it to compile.

[emard]

HI Thanks for fitting up the code - would you also like to try it? I initially started it on ESP8266 JTAGing to lattice XP2. It worked from SPIFFS but unreliable, often dropping the wifi connection or rebooting. Later the streaming version was developed on ESP32 which buffers packetized SVF stream coming from wifi and on-the fly JTAGs it after suffictient data arrive for one complete SVF command. Such web-based JTAG upload has some chance to work for 8266. It's up to SVF generating application to generate SVF file with multiple short commands nstead of one long single SVF command, because esp8266/esp32 can't buffer e.g. 4MB SVF content. approximately speaking, one SVF cmd shoudn't be longer than 1-2K) i Lattice has suitable option for such SVF but altera and xilinx don't. They generate all in single command. Such SVF must be first postprocessed by 3rd party script or application before seding to esp.

…

On 7/8/18, udif ***@***.***> wrote: I just left PR #1 to start this effort. My ultimate target (I would love to hear your feedback) is to be able to reuse old ESP8266-01 for JTAG programming via WiFi. I don't want to store any bitstream file on the SPIFFS, just send it to the FPGA on-the-fly. If I need to slow it a bit to get it to work reliably with WiFi, so be it. Is this feasible with the single core ESP8266? As I want to be able to reuse tons of old modules lying around, I really hope to be able to squeeze it into 512KB SPI memories. Currently websvf compiles to 373,160 bytes , hopefully leaving room for a small SPIFFS just for the minimal read-only web files. As I haven't used ESP8266 in the few years, I need to do some catch-up on SPIFFS and other ESP8266 stuff, so I haven't ran this code yet other than getting it to compile. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #2

[udif]

I spent some time reading the Xilinx XAPP503 dealing with SVF and XSVF and I see several approaches):

1. Allow only XSVF commands with a limited record length, and stop the uploader if such a command is detected.
2. Start with an SVF file, and use the Xilinx SDF2XSVF utility to generate an XSVF file with a limited record length
3. Preprocess the XSVF file, and do all the magic in the uploader, or in a separate preprocessor.
   Using the uploader is OK when you do command line processing, but if you want to upload an XSVF by a drag-and-drop operation into a web page I don't know if its possible (can you do the local preprocessing using Javascript on the browser, or is the file always uploaded as-is?)

The preprocessing is easy for the XSDRTDO* commands and can simply interleave the TDIvalue and TDOexpected bits, saving us from the need to save the entire TDI vector before shifting it in.
The XTDOMASK command is more problematic since its value has no length limit and needs to be used on all future TDO operations. The only way to cancel the buffering is to add an explicit TDO mask to every XSDRTDO* command preprocessed, potentially increasing the bitstream length.

I also understand that XSETSDRMASK and XSDRINC commands are deprecated and not used even by ISE (Let alone Vivado).

These tools from OpenOCD might help (svf2xsvf.py and xsvfdump.py)
https://github.com/arduino/OpenOCD/tree/master/contrib/xsvf_tools
This one is based on libXSVF:
https://github.com/makestuff/dump-xsvf

I will take a deeper look into libXSVF code and see how it works.

[udif]

Using the OpenOCD contrib xsvfcdump.py I checked one of my XSVF files generated by ISE 14.7 and it seems that the maximum SDR length in the file is 1024 (128 bytes).
For the moment I will go with this as a hard limit and see if I can get this to work by streaming the XSVF directly to JTAG.

[emard]

HI If your desired target is xilinx device with xsvf (and not svf) format, it's not so easy It would be best if there's some open source svf2xsvf application so we can fine tune output format suitable for small esp32/8266 buffering I've modified original cliffordwolf's svf parser to accept packetized data stream. Data from wifi come in form of packets, normally sequential from start to end of file. Each packet is passed as whole to the modified svf parser. When it receives sufficient data that enclose one complete svf command then it will execute it over jtag. Something similar has to be done with XSVF parser - currently I think it is not completely suitable because it originaly is done as old code, to receive char-by-char stream. Looking at how it is done for SVF code can similar be done for XSVF.

…

On 7/9/18, udif ***@***.***> wrote: I spent some time reading the Xilinx XAPP503 dealing with SVF and XSVF and I see several approaches): 1. Allow only XSVF commands with a limited record length, and stop the uploader if such a command is detected. 2. Start with an SVF file, and use the Xilinx SDF2XSVF utility to generate an XSVF file with a limited record length 3. Preprocess the XSVF file, and do all the magic in the uploader, or in a separate preprocessor. Using the uploader is OK when you do command line processing, but if you want to upload an XSVF by a drag-and-drop operation into a web page I don't know if its possible (can you do the local preprocessing using Javascript on the browser, or is the file always uploaded as-is?) The preprocessing is easy for the XSDRTDO* commands and can simply interleave the TDIvalue and TDOexpected bits, saving us from the need to save the entire TDI vector before shifting it in. The XTDOMASK command is more problematic since its value has no length limit and needs to be used on all future TDO operations. The only way to cancel the buffering is to add an explicit TDO mask to every XSDRTDO* command preprocessed, potentially increasing the bitstream length. I also understand that XSETSDRMASK and XSDRINC commands are deprecated and not used even by ISE (Let alone Vivado). These tools from OpenOCD might help (svf2xsvf.py and xsvfdump.py) https://github.com/arduino/OpenOCD/tree/master/contrib/xsvf_tools This one is based on libXSVF: https://github.com/makestuff/dump-xsvf I will take a deeper look into libXSVF code and see how it works. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #2 (comment)

[emard]

128 bytes is small enough. Actualy 1024 bytes would be still ok. Idea is not to buffer too many ethernet packets of 1492 bytes before JTAGing because JTAG processing by bitbang will accupy ESP CPU which may miss CPU cycles required to keepalive wifi connection

…

On 7/9/18, The EMARD ***@***.***> wrote: HI If your desired target is xilinx device with xsvf (and not svf) format, it's not so easy It would be best if there's some open source svf2xsvf application so we can fine tune output format suitable for small esp32/8266 buffering I've modified original cliffordwolf's svf parser to accept packetized data stream. Data from wifi come in form of packets, normally sequential from start to end of file. Each packet is passed as whole to the modified svf parser. When it receives sufficient data that enclose one complete svf command then it will execute it over jtag. Something similar has to be done with XSVF parser - currently I think it is not completely suitable because it originaly is done as old code, to receive char-by-char stream. Looking at how it is done for SVF code can similar be done for XSVF. On 7/9/18, udif ***@***.***> wrote: > I spent some time reading the Xilinx XAPP503 dealing with SVF and XSVF and > I > see several approaches): > > 1. Allow only XSVF commands with a limited record length, and stop the > uploader if such a command is detected. > 2. Start with an SVF file, and use the Xilinx SDF2XSVF utility to > generate > an XSVF file with a limited record length > 3. Preprocess the XSVF file, and do all the magic in the uploader, or in > a > separate preprocessor. > Using the uploader is OK when you do command line processing, but if you > want to upload an XSVF by a drag-and-drop operation into a web page I > don't > know if its possible (can you do the local preprocessing using Javascript > on > the browser, or is the file always uploaded as-is?) > > The preprocessing is easy for the XSDRTDO* commands and can simply > interleave the TDIvalue and TDOexpected bits, saving us from the need to > save the entire TDI vector before shifting it in. > The XTDOMASK command is more problematic since its value has no length > limit > and needs to be used on all future TDO operations. The only way to cancel > the buffering is to add an explicit TDO mask to every XSDRTDO* command > preprocessed, potentially increasing the bitstream length. > > I also understand that XSETSDRMASK and XSDRINC commands are deprecated > and > not used even by ISE (Let alone Vivado). > > These tools from OpenOCD might help (svf2xsvf.py and xsvfdump.py) > https://github.com/arduino/OpenOCD/tree/master/contrib/xsvf_tools > This one is based on libXSVF: > https://github.com/makestuff/dump-xsvf > > I will take a deeper look into libXSVF code and see how it works. > > > -- > You are receiving this because you commented. > Reply to this email directly or view it on GitHub: > #2 (comment)

[udif]

Here is my progress for the last few days:
Compiled and ran some code on ESP01. A modified websvf version, where the actual XSVF code is not called, only some Serial.printf() to see what we get from the client.
For the final version, I intend to disable Serial output and use those pins for TDI/TDO (I can hardly get 4 PIOs on the ESP01).
I may even be able to combine Serial output with JTAG if TCK won't change while serial data is driven out.

I also spent some time working on the (X)SVF code.
I created a new test, xsvftool-test that allows me to capture a JTAG log of XSVF/SVF from either a file or a packetized stream. The TDO and packet size are controlled by 2 separate random streams with different seeds, so they are not affected by each other. This means I can generate several runs with different packet sizes (or no packets at all), and yet generate the same random TDO sequence.
At the moment only 3 combinations are working - SVF from file or packets, and XSVF from files.
Adding the last (packetized XSVF) will be the major work needed.
However, By using this test I will be able to verify that the packetization did not break the JTAG sequences.

I cannot simply use SVF because ISE 14.7 generates SVF files with a single SDR command for the entire FPGA configuration bitsream. Only the XSVF output uses smaller data blocks (1024 bytes).

Also , there is a log of code duplication between the packetized and original SVF drivers.
I will try to factor that out to reduce code size and simplify it. The new xsvftool-test will let me regress any changes so I won't break the code.

For code, see https://github.com/udif/LibXSVF-ESP/tree/add_tests

[udif]

Also , there is a log of code duplication between the packetized and original SVF drivers.
I will try to factor that out to reduce code size and simplify it. The new xsvftool-test will let me regress any changes so I won't break the code.

Done. Liberally turned some auto vars into static.
I assume that after each upload I may need to call ESP.restart() so that each upload gets a fresh copy of all the static vars.

[emard]

HI Glad to hear about this great progress with XSVF! If you'd like to clean up the code please do so! I might like to check after this new changes how the code work for my ESP32-lattice boards. I think I have small spartan-6 board and nodemcu's around so I could assemble a setup to test XSVF too ESP01 has very few pin count, great if you can fit JTAG there :) for this I prefer ose of ESP12E in breakout modules or nodemcu 0.9/1.0 What do you think, can ESP SPI or some other DMA be used as accelerator for JTAG, in many long JTAG traansfer signals look similar to SPI transfer when TMS is not changing.

…

On 7/17/18, udif ***@***.***> wrote: > Also , there is a log of code duplication between the packetized and > original SVF drivers. > I will try to factor that out to reduce code size and simplify it. The new > xsvftool-test will let me regress any changes so I won't break the code. > Done. Liberally turned some auto vars into static. I assume that after each upload I may need to call ESP.restart() so that each upload gets a fresh copy of all the static vars. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #2 (comment)

[udif]

Quick update:
I've heavily modified libxsvf_xsvf() on my branch to work with streaming data.
The getbyte() function is now allowed to return -2 to indicate data not yet available.
In this case libxsvf_xsvf() will also return -2, and you are expected to subsequently call it again with the same parameters once new data is available.
The price I had to pay for this change was to make the code ugly and non-reentrant, full of static vars.
I definitely want to clean it further, as there is lots of duplication that can be moved into the READ_xx() macros.
It is also possible to move all the static vars to a context struct, giving us back the reentrancy.
All of this was tested on the host (linux) only.
I now need to integrate this with the esp8266 example and test with real hardware.
Relevant branch is:
https://github.com/udif/LibXSVF-ESP/commits/add_tests

Regarding your comment on the DMA engine, I'm familiar with this. The issues with using DMA is that it uses a fixed pinout requiring me to either pull wires directly from the chip and not the existing 4 IOs on the ESP01 connector.
As for whether the performance increase will be worth it (if at all - DMA setup also takes time) remains to be seen. This will definitely wait until I have something simple working with bit banging.

[emard]

HI Yes the function has to be repeatedy called any number times, in each time supplying one ethernet packet from the data stream. I think also my svf edition is full of static vars, ugly and non-reentrant too. Hopfully not too buggy also. I think ESP doesn't need to reenter at all (no recursion, no linux, no threads) all it has to do is process one packet as fast as possible with buffering, parsing and JTAG-ing and return control to outside loop for uninterrupted wifi connection. Also this ESP realloc is suspicious to me as it probably affects stability and causes ESP reboots; frequency of calling realloc should be reduced.

…

On 7/22/18, udif ***@***.***> wrote: Quick update: I've heavily modified libxsvf_xsvf() on my branch to work with streaming data. The getbyte() function is now allowed to return -2 to indicate data not yet available. In this case libxsvf_xsvf() will also return -2, and you are expected to subsequently call it again with the same parameters once new data is available. The price I had to pay for this change was to make the code ugly and non-reentrant, full of static vars. I definitely want to clean it further, as there is lots of duplication that can be moved into the READ_xx() macros. It is also possible to move all the static vars to a context struct, giving us back the reentrancy. All of this was tested on the host (linux) only. I now need to integrate this with the esp8266 example and test with real hardware. Regarding your comment on the DMA engine, I'm familiar with this. The issues with using DMA is that it uses a fixed pinout requiring me to either pull wires directly from the chip and not the existing 4 IOs on the ESP01 connector. As for whether the performance increase will be worth it (if at all - DMA setup also takes time) remains to be seen. This will definitely wait until I have something simple working with bit banging. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #2 (comment)

[udif]

I haven't implied that ESP needs reentrant code.
Given the low memory capacity and the fact that we a single task run bare-metal program, non reentrant code is OK.
However, what I meant was that the original code in LibXSVF was written to be much cleaner -
All variables are local, and all the state is encapsulated in the libxsvf_host structure.
This is OOP done in C, with libxsvf_host as the class, and the function pointers instead of virtual functions.
You could run 2 parallel JTAG interface using the same library and different libxsvf_host structures.
Ofcourse, this is irrelevant for the ESP.

In the meantime, I did some cleanup on libxsvf_xsvf() and moved the retry logic into trhe READ_xxx() macros, making the code much simpler.

[emard]

Yes indeed, original code was much cleaner. Looking in this direction, moving this static vars to a struct like libxsvf_host might make it reentrant again

…

On 7/22/18, udif ***@***.***> wrote: I haven't implied that ESP needs reentrant code. Given the low memory capacity and the fact that we a single task run bare-metal program, non reentrant code is OK. However, what I meant was that the original code in LibXSVF was written to be much cleaner - All variables are local, and all the state is encapsulated in the libxsvf_host structure. This is OOP done in C, with libxsvf_host as the class, and the function pointers instead of virtual functions. You could run 2 parallel JTAG interface using the same library and different libxsvf_host structures. Ofcourse, this is irrelevant for the ESP. In the meantime, I did some cleanup on libxsvf_xsvf() and moved the retry logic into trhe READ_xxx() macros, making the code much simpler. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #2 (comment)