bandwidth test version - iguana for unix

[jl777]

I wouldnt say "easy", but I will do the work to make it happen so it will seem easy to the users. Also unlike the typical torrent delivered bootstrap files, iguana bundles are never changing, so the normal bittorrent network can be used and the entire bittorrent peers acts as permanent archival nodes.

Additionally, the bundles are fully verified and not just accepted via checkpoint

 

[jl777]

I fixed things:

BTC u.0 b.0 v.0/0 (1915 1st.110) to 121 N[202] Q.3883 h.403972 r.403972 c.616.1MB s.233947 d.103 E.103:269289 est.19 7.05GB 0:05:05 4.262 peers.76/256 Q.(0 0) L.403972 M.403971 0000000000000000057a72b0619c56e80298accd8fed0432bb9d30e7d2c14cb1

BTC u.0 b.0 v.0/0 (1969 1st.148) to 144 N[202] Q.5991 h.403972 r.403972 c.1.58GB s.307676 d.135 E.135:347436 est.10 17.92GB 0:10:02 1.276 peers.64/256 Q.(0 0) L.403972 M.403971 0000000000000000057a72b0619c56e80298accd8fed0432bb9d30e7d2c14cb1

BTC u.0 b.0 v.0/0 (1926 1st.181) to 170 N[202] Q.31110 h.403974 r.403974 c.2.32GB s.366418 d.168 E.168:411843 est.3 33.34GB 0:20:07 6.832 peers.59/256 Q.(0 1546) L.403974 M.403973 000000000000000004f48ae761f7b51b48e1b4224419922e3c90d8e753de4d20

in about 30 minutes all blocks downloaded

BTC u.201 b.169 v.169/201 (1978 1st.201) to 201 N[202] Q.17243 h.403978 r.403978 c.736.6MB s.403978 d.201 E.201:4 est.64 736.6MB 0:05:23 999.445 peers.13/8 Q.(0 0) L.403978 M.403977 000000000000000000fc8f6bbad5629999cbb3d192a95d5a8ad5a8c8fcc2ad31

took 5 minutes to generate the 201 spends vectors u.201 and 169 balance files. but since the raw data is bigger than system RAM, things slow down at this point...
I could do a staged multistep process, but since it will complete in about an hour i wont bother to speed this part up as a 64GB machine would blaze right through this

 

[jl777]

it bothered me, the known slowness, so I made a few tweaks and after half a dozen more sync tests, I got:

BTC u.112 b.110 v.0/112 (1997 1st.140) to 153 N[203] Q.1498 h.404054 r.404054 c.1.21GB s.305660 d.137 E.137:336152 est.17 17.86GB 0:10:06 4.723 peers.63/256 Q.(0 0) L.404054 [202:53] M.404053 000000000000000001125311bb549d06272a0a1e88dc5f8161084a0cf96e5d3c

305660 blocks synced in 10 minutes and saved 137 bundles, 274,000 blocks worth. lag of about 25,000 blocks. 112 spendvectors and 110 balance files

BTC u.158 b.154 v.0/158 (1920 1st.174) to 183 N[203] Q.4212 h.404055 r.403975 c.11.64GB s.376708 d.170 E.170:409070 est.14 41.16GB 0:20:00 257.018 peers.61/256 Q.(5277 65760) L.404055 [202:54] M.404054 00000000000000000139abcd83383600e84ca46b06851107c88eb41431898699

376708 blocks synced in 20 minutes and 170 bundles saved, 340000, lag of about 35000 blocks. 158 spendvectors and 154 balances, so they are lagging a bit, but not too bad

BTC u.181 b.165 v.0/181 (56 1st.202) to 199 N[203] Q.7654 h.404056 r.404056 c.1.17GB s.404056 d.196 E.196:437604 est.4 55.18GB 0:35:35 9.168 peers.38/8 Q.(0 0) L.404056 [202:55] M.404055 000000000000000000b2211355bee045f3be8cc7f61362a475310e90fd934824

finished downloading and 196 bundles already completed. I think that is the best result so far. only lagging by 12000 blocks. 181 spendvectors and 165 balancefiles

The problem seems to be not having SSD and things are thrashing... Looks like it will be another half hour to finish the rest of the balance files, but the mostly idle CPU can be doing sig validations during the idle time

I have to change things to be height based for the volatile data generation, that way the same code can be used for historical and realtime calcs. I must have done almost 500 BTC syncs this year!

 

[jl777]

WARNING iguana_hash2set overwrite [blockadd] 0000000000000000014fc67e896071d9cc5345fea4d04795ff6ee8f9ae142268 with 00000000000000000533c18c743dc45dd7526c7190996a49aee2b6fd3f7c104b [202:145]

thinking the above was a bug, i looked them up in blockchain.info
turns out the first one was orphaned and the second one is mainchain
so iguana is picking up these small reorgs and it handled this one properly

 

[jl777]

latest version on fast connection VPS downloads the full blockchain in about 30 minutes, takes another 15 minutes to generate the rest of the read only dataset. All during this time the CPU usage is close to maximum. At this point, there is a serial process for updating the volatile data. While it could be done in parallel, since they access potentially the same data, locking would need to be done, or merging and that is a more work

with somewhat dated CPU and 8 cores, it seems to be about one hour of a single core going close to 100%, which means it really cant be sped up too much. This leave 7 cores idle for about half hour, which is perfect for doing signature validation, so probably wont make sense to optimize the balance calcs.

Now why does it take so long to process all this?

I am seeing about 6 million updates per bundle, the recent ones, so that is literally hundreds of millions of updates, just for the utxo! I think the total "DB" operations is on the order of a billion operations.

BTC.RT404000 u.202 b.202 v.202/206 (170/170 1st.202) to 202 N[203] h.404170 r.404170 c.705.0MB s.404170 d.202 E.202:461495 est.4 54.36GB peers.130/1024 Q.(0 0) L.405351 [202:169] M.404169 000000000000000001ef7cfcec377af174fc0bf552f4f669f6bb85ab98686af0 bQ.-4 1:39:53 153.553

 

[jl777]

I tried a non-serialized settings and got a horrible 2:14 hr time to generate the entire dataset:

BTC.RT0 u.202 b.202 v.202/202 (406/406 1st.202) to 202 N[203] h.404406 r.404406 c.687.7MB s.404406 d.202 E.202:474238 est.3 50.72GB peers.19/8 Q.(0 0) L.404406 [202:405] M.404405 00000000000000000270e3a9b059c55837995679536595afbca47a3c52ee594f bQ.-219 2:14:49 98.210

At this point I force and exit as there is a bug with immediately doing the realtime blocks, ie it takes another hour, when it really only should take less than a minute:

BTC.RT404406 u.202 b.202 v.202/0 (406/406 1st.202) to 202 N[203] h.404406 r.404406 c.687.7MB s.404406 d.202 E.202:2 est.64 687.7MB peers.121/1024 Q.(0 0) L.405538 [202:405] M.404405 00000000000000000270e3a9b059c55837995679536595afbca47a3c52ee594f bQ.1 0:01:25 124.885

above 1 minute 25 seconds is due to the first time verification of the SHA256 for the volatiledata. The second time is faster:

BTC.RT404407 u.202 b.202 v.202/0 (407/407 1st.202) to 202 N[203] h.404407 r.404407 c.688.0MB s.404407 d.202 E.202:2 est.64 688.0MB peers.122/1024 Q.(0 0) L.405539 [202:406] M.404406 00000000000000000276502b29226a34dcb84f441f8235c7b6368635217d9b5d bQ.1 0:00:29 118.482

29 seconds! one second below my 30 second target. Had to fix bugs yesterday as things were not quite right, but now I am coding up the combined RT/historical queries and hope to get them stable this weekend.

 

[pebwindkraft]

Hi James,

you say:

But currently a lot more space is needed during sync, ~100GB

wouldn't this be an exit criteria for IoT, as grewalsatinder requested?
I was actually thinking to bring this to a Raspberry Pi2 (4 Core, 1 Gig Mem, 32/64Gig SD card as HDD). I had setup my small RPi2 to monitor BitCoin 12 client behavior, and found, that during re-sync the CPU is constantly writing with 5-12 MB/sec data to the SD card. Modern SD cards shouldn't be affected too much (older got corrupt). With 1 Gig of Memory there is also not "lots of space" for a RAM disk. But it has a nice 4core CPU (@900 Mhz), and one could easily measure the effects

rgds,
Volker

 

[jl777]

You can reduce the space needed by not going all out parallel, but realistically the low end nodes will have a hard time keeping up with the full blockchain. as it is, the normal desktops are not really practical for a full node, so the first step is to fix that.

then we can explore what it takes to make a pruning iguana

 

[pebwindkraft]

just gave it a try on my old 2011 13" MacbookPro with OSX 10.10.5 (Yosemite). It has 4Gig RAM, and an Intel Core i5:

Loretta:/Users/volker # git clone https://github.com/jl777/SuperNET
Cloning into 'SuperNET'...
...
Checking out files: 100% (1293/1293), done.

Loretta:/Users/volker # cd SuperNET/
Loretta:/Users/volker/SuperNET # ./m_onetime m_unix
Cloning and Building secp256k1 library...
Cloning into 'secp256k1-zkp'...
remote: Counting objects: 3528, done.
remote: Total 3528 (delta 0), reused 0 (delta 0), pack-reused 3528
Receiving objects: 100% (3528/3528), 1.41 MiB | 796.00 KiB/s, done.
Resolving deltas: 100% (2534/2534), done.
Checking connectivity... done.
./autogen.sh: line 3: autoreconf: command not found
./m_onetime: line 9: ./configure: No such file or directory
make: *** No targets specified and no makefile found. Stop.
cp: .libs/libsecp256k1.a: No such file or directory
Already up-to-date.

So I can see libsecp256k1 is prepared - cool. I need to dig into autogen, not sure what's going on. Maybe need some homebrew software. For sure next steps failed:

Loretta:/Users/volker/SuperNET # cd iguana
Loretta:/Users/volker/SuperNET/iguana # ./m_unix
Already up-to-date.
clang: error: no such file or directory: '../includes/libsecp256k1.a'
Loretta:/Users/volker/SuperNET/iguana #

 

[jl777]

you need to install autotools, libtool, autoconf and gmp, all to get the libsecp256k1 lib built.

once the libsecp256k1 lib has all its dependencies, it should just build.

to start a sync you can use the genbtc or btcd script

 

[pebwindkraft]

correct.

I have downloaded and installed iguana again (Monday, 18th of April 2016).
System is a MBPro 13“ from 2011, with an Intel i5@2.3Ghz and 4Gig RAM, running on Yosimete (10.10.5).

first observation:
CPU usage is at 40%, and memory of the iguana process is like 350 MBytes.
Disk I/O is constantly 5MBytes/sec, only „writing“
Net I/O is in the 1100KBytes/sec (I have a 20mbit DSL line)
Looking at the output from the top command, loadavg is above 1.0, which in my opinion means, there are more processes waiting in the queue, as the system can process. Might be disk I/O…
Temperature of my Mac varied between 75 and 92 degrees Celsius, with fans running between 4000 and 6200 rpm.

top:
====
Processes: 205 total, 4 running, 12 stuck, 189 sleeping, 891 threads 09:34:26
Load Avg: 2.41, 3.02, 2.41 CPU usage: 41.64% user, 39.95% sys, 18.40% idle
SharedLibs: 15M resident, 13M data, 0B linkedit.
MemRegions: 24666 total, 1736M resident, 104M private, 320M shared.
PhysMem: 3896M used (1001M wired), 199M unused.
VM: 514G vsize, 1063M framework vsize, 0(0) swapins, 0(0) swapouts.
Networks: packets: 557560/776M in, 476140/40M out.
Disks: 71949/1884M read, 141349/3094M written.

PID COMMAND %CPU TIME #TH #WQ #PORT MEM PURG CMPR PGRP PPID STATE
3012 mdworker 6.9 00:00.22 6 2 56 2544K- 0B 0B 3012 1 sleeping
3004 AppleSpell 0.0 00:00.03 2 0 42 1316K 0B 0B 3004 1 sleeping
3001 mdworker 4.7 00:00.81 4 0 52 2964K- 0B 0B 3001 1 sleeping
2998 mdworker 5.6 00:01.01 4 0 52 3344K- 0B 0B 2998 1 sleeping
2997 mdworker 4.0 00:01.03 4 0 52 3456K- 0B 0B 2997 1 sleeping
2985 ocspd 0.0 00:00.01 1 0 19 992K 0B 0B 2985 1 sleeping
2968 com.apple.iC 0.0 00:00.16 2 0 45 2116K 0B 0B 2968 1 sleeping
2966 top 3.1 00:08.65 1/1 0 26 2396K 0B 0B 2966 664 running
2930 systemstats 0.0 00:00.32 2 1 34 3400K 0B 0B 2930 1 sleeping
2928 iguana 238.9 19:55.30 72/4 0 168 373M 0B 0B 2928 665 running
2571 Activity Mon 0.0 00:17.73 7 4 209 20M- 9620K+ 0B 2571 1 sleeping
2398 DashboardCli 0.0 00:00.79 10 0 186 16M 600K 0B 291 2
…


top (20min later):
==================
Processes: 199 total, 3 running, 7 stuck, 189 sleeping, 784 threads 09:53:15
Load Avg: 3.70, 3.40, 2.97 CPU usage: 41.89% user, 33.56% sys, 24.53% idle
SharedLibs: 15M resident, 13M data, 0B linkedit.
MemRegions: 22949 total, 1374M resident, 102M private, 259M shared.
PhysMem: 3486M used (1001M wired), 608M unused.
VM: 499G vsize, 1064M framework vsize, 0(0) swapins, 0(0) swapouts.
Networks: packets: 1562239/2181M in, 1213037/93M out.
Disks: 116136/2311M read, 322395/7847M written.
…

—> I stoppped the iguana process, changed ulimit, and restarted. 5min later:

Processes: 200 total, 3 running, 7 stuck, 190 sleeping, 800 threads 10:15:06
Load Avg: 6.73, 5.38, 4.12 CPU usage: 36.27% user, 6.61% sys, 57.10% idle
SharedLibs: 15M resident, 13M data, 0B linkedit.
MemRegions: 23408 total, 1730M resident, 97M private, 312M shared.
PhysMem: 3630M used (1022M wired), 464M unused.
VM: 502G vsize, 1064M framework vsize, 0(0) swapins, 0(0) swapouts.
Networks: packets: 2110756/2896M in, 1642397/138M out.
Disks: 164149/3764M read, 554589/12G written.

PID COMMAND %CPU TIME #TH #WQ #PORT MEM PURG CMPRS PGRP PPID STATE
3357 CVMCompiler 0.0 00:00.24 2 1 28 13M 0B 0B 3357 1 sleeping
3354 top 2.1 00:01.79 1/1 0 20 2260K 0B 0B 3354 664 running
3318 mdworker 0.0 00:00.63 4 0 52 6356K 0B 0B 3318 1 sleeping
3314 mdworker 0.0 00:00.62 4 0 52 6076K 0B 0B 3314 1 sleeping
3305 mdworker 0.0 00:00.71 4 0 52 6168K 0B 0B 3305 1 sleeping
3303 mdworker 0.0 00:00.63 4 0 52 6148K 0B 0B 3303 1 sleeping
3251 iguana 144.4 14:07.67 96/1 0 189 715M+ 0B 105M 3251 665 running
3004 AppleSpell 0.0 00:00.03 2 0 42 52K 0B 1264K 3004 1 sleeping
2930 systemstats 0.0 00:00.48 2 1 34 2764K 0B 1124K 2930 1 sleeping
2571 Activity Mon 6.5 01:01.88 7 4 214+ 17M+ 7696K- 5808K 2571 1 sleeping

no more stuck processes, loadavg going higher… then going back to run around 4.
The whole system remains responsive, a bit slow but responsive
du -h -s in the DB and tmp directory of Supernet takes longer than normal (very much longer!)


Observations:
=============
After 45min I had it collected 3.1 Gig, and the iguana/tmp dir was having like 1.5Gig.

Also I saw this invalid pubkey somewhere:
>>
>> BTC BLOCKS.349 RECV 651.777kb ave 1912.4 | dup.0 0.000kb after.0 0.000kb 6.134kb/sec 100.00%
>> BTCD BLOCKS.489 RECV 5541.067kb ave 11603.4 | dup.0 0.000kb after.0 0.000kb 271.627kb/sec 10.52% SLOW
>> multisig.1 of 2: invalid pubkey[20] len -1
>> multisig.1 of 2: invalid pubkey[16] len -1
>> multisig.1 of 2: invalid pubkey[1c] len -1
>>
not sure, what it means though …


After 1 hour I would arrive somewhere here:
>>>>>>>>>>>>>> EMIT.[467] BTCD | 1st.19 h.500 c.0 s.[ 2] maxB.500 NET.(h0 b592) 25:03
>>>>>>>>>>>>>> EMIT.[558] BTCD | 1st.19 h.500 c.0 s.[ 2] maxB.500 NET.(h0 b675) 25:15
BTCD BLOCKS.153392 RECV 1.09GB ave 7638.0 | dup.8442 16.4MB after.28 13.198kb 42.710kb/sec 41.20%
BTC.RT0 u.0 b.0/0 v.0 (0+986/2000 1st.29).s0 to 203 N[204] h.407816 r.64028 c.64000 s.122017 d.0 E.32 maxB.327 peers.56/128 Q.(2028 0) (L.407816 203:1816) M.407815 0000000000000000049bf7b10ed417574ddae39c14a6fc37bb9771d0ab8ff78f bQ.2173 0:28:05 stuck.1226 max.1226
>>>>>>>>>>>>>> EMIT.[418] BTCD | 1st.19 h.500 c.0 s.[ 2] maxB.500 NET.(h0 b720) 25:23
BTC.RT0 u.0 b.0/0 v.0 (0+986/2000 1st.29).s0 to 203 N[204] h.407816 r.64028 c.64000 s.122017 d.0 E.32 maxB.327 peers.56/128 Q.(1014 0) (L.407816 203:1816) M.407815 0000000000000000049bf7b10ed417574ddae39c14a6fc37bb9771d0ab8ff78f bQ.2172 0:28:13 stuck.1234 max.1234
BTCD BLOCKS.154398 RECV 1.09GB ave 7591.9 | dup.8618 16.5MB after.28 13.198kb 33.921kb/sec 37.35%
>>>>>>>>>>>>>> EMIT.[ 13] BTCD | 1st.19 h.500 c.0 s.[ 2] maxB.500 NET.(h0 b-175) 25:41
BTCD BLOCKS.155410 RECV 1.09GB ave 7546.3 | dup.8623 16.5MB after.28 13.198kb 36.875kb/sec 33.91%
BTC.RT0 u.0 b.0/0 v.0 (0+986/2000 1st.29).s0 to 203 N[204] h.407816 r.64028 c.64000 s.122017 d.0 E.32 maxB.327 peers.56/128 Q.(2028 0) (L.407816 203:1816) M.407815 0000000000000000049bf7b10ed417574ddae39c14a6fc37bb9771d0ab8ff78f bQ.2171 0:28:35 stuck.1256 max.1256
Not sure what the EMIT means. Calculating the processed amount and size of disk usage, it would take +6 hours to download the whole blockchain. I’ll observe it a bit…


A Note on Installation:
=======================
Though I had xcode installed, aclocal didn’t work. I installed homebrew, and then:
# brew install autoconf
# brew install automake
# brew install gmp

2.) libsecp256
it complained, that libsecp256 was not there in includes, so I linked it.
Loretta:/Users/volker/SuperNET/includes # ln -s ../osx/libsecp256k1 .

3.) I had to change ulimit
During the syncing, I have many, many messages like this:
>>
>> cant create.(tmp/BTC/252000/.tmpmarker) errno.24 Too many open files
>> cant create.(tmp/BTC/18000/.tmpmarker) errno.24 Too many open files
>>
Loretta:/Users/volker/SuperNET # ulimit -n 2048

 

[pebwindkraft]

A note on the used installation:

4.) iguana
# ./m_onetime m_unix
Cloning and Building secp256k1 library...
fatal: destination path 'secp256k1-zkp' already exists and is not an empty directory.
configure.ac:20: installing 'build-aux/compile'
configure.ac:5: installing 'build-aux/config.guess'
configure.ac:5: installing 'build-aux/config.sub'
configure.ac:9: installing 'build-aux/install-sh'
configure.ac:9: installing 'build-aux/missing'
Makefile.am:3: error: Libtool library used but 'LIBTOOL' is undefined
Makefile.am:3: The usual way to define 'LIBTOOL' is to add 'LT_INIT'
Makefile.am:3: to 'configure.ac' and run 'aclocal' and 'autoconf' again.
Makefile.am:3: If 'LT_INIT' is in 'configure.ac', make sure
Makefile.am:3: its definition is in aclocal's search path.
Makefile.am: installing 'build-aux/depcomp'
parallel-tests: installing 'build-aux/test-driver'
autoreconf: automake failed with exit status: 1
checking build system type... x86_64-apple-darwin14.5.0
checking host system type... x86_64-apple-darwin14.5.0
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... build-aux/install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
./configure: line 2887: LT_INIT: command not found
checking whether make supports nested variables... (cached) yes
./configure: line 2929: PKG_PROG_PKG_CONFIG: command not found
checking for ar... /usr/bin/ar
checking for ranlib... /usr/bin/ranlib
checking for strip... /usr/bin/strip
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking for gcc... gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking dependency style of gcc... (cached) gcc3
checking how to run the C preprocessor... gcc -E
checking for gcc option to accept ISO C89... (cached) none needed
checking for brew... /usr/local/bin/brew
checking if gcc supports -std=c89 -pedantic -Wall -Wextra -Wcast-align -Wnested-externs -Wshadow -Wstrict-prototypes -Wno-unused-function -Wno-long-long -Wno-overlength-strings... yes
checking if gcc supports -fvisibility=hidden... yes
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for __int128... yes
checking for __builtin_expect... yes
checking for __builtin_clzll... yes
checking for x86_64 assembly availability... no
checking gmp.h usability... no
checking gmp.h presence... no
checking for gmp.h... no
checking openssl/crypto.h usability... yes
checking openssl/crypto.h presence... yes
checking for openssl/crypto.h... yes
checking for main in -lcrypto... yes
checking for EC functions in libcrypto... yes
checking whether byte ordering is bigendian... no
configure: Using assembly optimizations: no
configure: Using field implementation: 64bit
configure: Using bignum implementation: no
configure: Using scalar implementation: 64bit
configure: Using endomorphism optimizations: no
configure: Building ECDSA pubkey recovery module: no
configure: ******
configure: WARNING: experimental build
configure: Experimental features do not have stable APIs or properties, and may not be safe for production use.
configure: Building ECDH module: yes
configure: Building Schnorr signatures module: yes
configure: Building range proof module: yes
configure: ******
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: error: cannot find input file: `Makefile.in'
make: *** No targets specified and no makefile found. Stop.
cp: .libs/libsecp256k1.a: No such file or directory
remote: Counting objects: 231, done.
remote: Compressing objects: 100% (163/163), done.
remote: Total 231 (delta 164), reused 56 (delta 56), pack-reused 12
Receiving objects: 100% (231/231), 443.37 KiB | 0 bytes/s, done.
Resolving deltas: 100% (164/164), completed with 12 local objects.
From https://github.com/jl777/SuperNET
37f503a..3955bda master -> origin/master
Updating 37f503a..3955bda
Fast-forward
crypto777/iguana_OS.c | 2 +
crypto777/nanosrc/transports/utils/base64.h | 4 +-
iguana/btcd | 3 +-
iguana/confs/BTCD_hdrs.h | 17 ++++-
iguana/iguana777.c | 57 ++++++++-------
iguana/iguana777.h | 4 +-
iguana/iguana_blocks.c | 4 +-
iguana/iguana_bundles.c | 18 +++--
iguana/iguana_init.c | 8 +-
iguana/iguana_ramchain.c | 83 ++++++++++++++-------
iguana/iguana_recv.c | 14 ++--
iguana/iguana_tx.c | 8 +-
iguana/iguana_unspents.c | 173 +++++++++++++++++++++++++++++---------------
iguana/main.c | 8 +-
14 files changed, 266 insertions(+), 137 deletions(-)
Loretta:/Users/volker/SuperNET #

 

[jl777]

thanks for the detailed results!
doing multiple coin syncs at the same time will use a lot of files now, but you can tweak the sync settings, especially via the startpend and endpend in the scripts.

I have them tuned for higher bandwidth machines and it defaulted to automatically syncing BTCD on osx.

I found an issue with realtime mode and on a new config, it is quite possible for there to be new issues as I wont have ever seen it run on that config. if it gets stuck, just ctrl-c and run it again,the data files are append only and cross verified so the worst that you need to do is delete a messed up file. I added various automatic deletes of bad files, but I am sure there are cases that I missed as it is possible for a power out at any point

 

[jl777]

i hope it is ok that I use part of your post in the readme.md for other users? Also does anything appear on http://127.0.0.1:7778 or 7779?

at 20mbps 6hrs is about as fast as you can download 65GB, at such slow speeds iguana shouldnt have a problem in keeping up (E.nnn would be close to 1st)

I am concerned about being able to complete the sync without errors on a low RAM setup, I havent optimized that yet.

On my monster server with 12 cores, I see 1200% CPU, 500MB/sec HDD, 80to 100MB/sec bandwidth. not sure of temp as it is a remote server, but my guess is you can cook breakfast on top of the CPUs

 

[pebwindkraft]

ok to use the info for readme.md, happy it is useful
yes, on http://127.0.0.1:7779 I see all options to play with the system (SuperNET 7777 GUI).
I'd like to create a raw transaction, and bring it to my cold storage... but therefor I'd need the "import_mster_pubkey"? Hint: the bitcoin RPCs should be sorted, or grouped? (I know, still alpha software).
Also I would like to use the DEX and buy some nxt. Interesting piece of work. Really.
monster server temps: getting hungry now

 

[jl777]

does the link in the upper left corner for 7777 GUI work?
iguana doesnt need to importprivkeys and you can just sign the transaction with the wif format privkey after creating the rawtransaction bytes.
however the RPC layer is just getting in place, so I wouldnt use it for anything real yet.

I will be pushing a version that publishes a bitcoin RPC compatible on the corresponding port. will make a new iguana RPC thread soon.

the API test page is just for testing the API, the actual usecases are intended to get GUI for them, like the one on port 7777. though that is just a quick and dirty ugly gui.

As soon as the realtime sync issues are solved, my next step is to get listunspents working at the RPC level and then the atomic cross chain is ready to resume debugging. I got to where I needed to have so many bitcoind's and wallets around to test it, I figured I should push forward to get iguanacore ready to handle multicoins and RPC compatibility. That way, just by running iguana I can do atomic swaps, etc.

I am happy to make sure whatever usecase you have in mind is working, so if things are not working, just post here. There are so many bitcoin RPC, I am sure many are not quite right and many are still stubs. the hard part is getting the core to work, but connecting it properly to the RPC layer is also needed

I really appreciate the testing! Hopefully more people will take the plunge into iguanaland

 

[pebwindkraft]

The link in the top left works. I can see shortly something about JSON (looks like the Pangea page...), and then the screen switches to a green top bar with these menu items:
Iguana Blockexplorer Coins Debug Peers Pangea Instandex Tradebot Bitmap Settings
and below a white area named "Block Explorer tab".

On the RPC layer: you're right, I'll start slowly. maybe connect to testnet first, before trying crazy stuff.

 

[jl777]

just checked and current version wont do so well with realtime mode for BTC, but the historical bundles are all very valid, in fact once they are verified they wont need to change.

to test the realtime mode, you would need to stay with BTCD for now. I would be happy to send you some coins for your testing
[doublepost=1461022348][/doublepost]I am using BTCD as the "testnet", but currently I only have verified the blockchain level RPC calls, so you can get blockhashes, block data, tx, etc, but not the transactions yet

just posted: https://bitco.in/forum/threads/iguana-rpc-docs-supernet-org.1097/

 

[pebwindkraft]

Hi James,

couldn't resist, but had to give it a try on a Raspberry.
At the end I have a clear understanding what happens with Bitcoin Core and Ethereum on this device, and if the usage of disk I/O reduces drastically, I should be able to see this also on the RPi2.
So I went again through the installation cycle (basically on OpenSUSE Tumbleweed). It is ARM architecture. I can go through the libsecp256k1, no prob. But there is some "pointer to integer conversion with different size" error in a later stage (looking into code, I'd need some idea were to look...). It comes up as warning, not as error. But later when starting iguana/m_unix, it would show the same:

iguana_tx.c: In Funktion »iguana_scriptdata«:
iguana_tx.c:30:22:Warnung: Typkonvertierung von Zeiger auf Ganzzahl anderer Breite [-Wpointer-to-int-cast]
fileptr[0] = (uint64_t)OS_mapfile(fname,&fileptr[1],0);
^
iguana_tx.c:35:32:Warnung: Typkonvertierung in Zeiger von Ganzzahl anderer Breite [-Wint-to-pointer-cast]
memcpy(scriptspace,(void *)(fileptr[0] + scriptpos),scriptlen);
^
iguana_unspents.c: In Funktion »iguana_alloctxbits«:
iguana_unspents.c:125:41:Warnung: Typkonvertierung in Zeiger von Ganzzahl anderer Breite [-Wint-to-pointer-cast]
int32_t tlen; uint8_t *TXbits = (uint8_t *)((long)ramchain->H.data + ramchain->H.data->T
^

iguana_unspents.c: In Funktion »iguana_alloccacheT«:
iguana_unspents.c:141:49:Warnung: Typkonvertierung in Zeiger von Ganzzahl anderer Breite [-Wint-to-pointer-cast]
int32_t i,tlen; struct iguana_txid *T = (void *)((long)ramchain->H.data + ramchain->H.da
^

 

[jl777]

thanks for the report!
pushed fixes, but I cant verify it worked. hopefully the warnings are gone

also, iguana is quite a bit more resource intensive during the parallel sync, ie 100% CPU, 100% bandwidth, 100% HDD usage

I havent optimized HDD usage during sync yet, so you will need ~100GB of HDD space for full sync. set "startpend" and "endpend" to lower values to reduce the amount of parallelism and resources used, if default settings are too big.