# Pastebin 73V4aGO4 [2020-04-15 04:49:52] Hello [2020-04-15 04:50:53] I'd like to contribute to the project, but would like to learn, if you are willing to accept my change, before I even start working. References: [2020-04-15 04:50:55] https://repo.getmonero.org/monero-project/ccs-proposals/-/merge_requests/138 [2020-04-15 04:51:09] https://repo.getmonero.org/monero-project/ccs-proposals/-/blob/b8033720941c69a1dd45c26bf6c1bb1d03863b77/mj-compil-time-reduction.md [2020-04-15 04:51:42] The POC contribution can be easily done unfunded. [2020-04-15 05:02:28] mjxmr: i think that some parts go a bit crazy with templates (-ftemplate-depth needs to be increased). do you think it would be worth going through with clang's profiler seeing if there's anything that should be changed? [2020-04-15 05:04:13] → Yasmeen65Pouros joined (~Yasmeen65@ns334669.ip-5-196-64.eu) [2020-04-15 05:04:43] asymptotically: I think, that using profilers is always a better idea, than using a gut feeling (same applies for speed optimisations). Otherwise a programmer falls into a dogmatic way of thinking, while I prefer to work evidence based. [2020-04-15 05:05:09] → TheoStorm joined (~TheoStorm@host-p8vu8h.cbn1.zeelandnet.nl) [2020-04-15 05:06:02] However in this case, boost headers, I'm absolutely sure, that this is the major slowdown. Abuse of templates create the need of having to include the boost headers in a parent header. This can be avoided with some effort. [2020-04-15 05:06:24] ⇐ TheoStorm quit (~TheoStorm@host-p8vu8h.cbn1.zeelandnet.nl): Remote host closed the connection [2020-04-15 05:06:56] I am so sure, because I ate my teeth on this many times already. [2020-04-15 05:08:32] also what does the ccache integration with cmake do? i thought you just had to set CC and CXX [2020-04-15 05:08:34] ⇐ Yasmeen65Pouros quit (~Yasmeen65@ns334669.ip-5-196-64.eu): Ping timeout: 240 seconds [2020-04-15 05:10:04] Yes, you can do this of course, but not everybody is proficient enough to do this. My goal would be to set the CC / CXX automatically if ccache is detected in path. This is minor thing, yeah. [2020-04-15 05:16:46] At our job we disable ccache for special final builds, to minimize the risk of ccache affecting the binaries that go to the customer, but we have it on by default. [2020-04-15 05:19:09] ... during the development, I mean, so 90% of time. [2020-04-15 05:19:15] → TheoStorm joined (~TheoStorm@host-p8vu8h.cbn1.zeelandnet.nl) [2020-04-15 05:28:08] → coppa joined (~Thunderbi@2a01:e0a:13c:6110:55e8:2142:245d:8808) [2020-04-15 05:31:34] → coppa1 joined (~Thunderbi@91-164-241-132.subs.proxad.net) [2020-04-15 05:31:36] ⇐ coppa quit (~Thunderbi@2a01:e0a:13c:6110:55e8:2142:245d:8808): Remote host closed the connection [2020-04-15 05:31:36] * coppa1 → coppa [2020-04-15 05:36:08] asymptotically: "Abuse of templates create the need of having to include the boost headers in a parent header. " A poor man's solution to this is using precompiled headers of the boost headers collection. This helps to reduce the compilation time by up to 50% per header, but a lot more this can be achieved via wrappers. [2020-04-15 05:39:38] ⇐ wizardsmoke quit (uid209547@gateway/web/irccloud.com/x-ixsbdjgfdwptbctr): Quit: Connection closed for inactivity [2020-04-15 05:40:49] Perhaps it wouldn't be a bad idea to start off with this, as the least intrusive measure? [2020-04-15 05:41:30] I’d wait for feedback by moneromooo / vtnerd [2020-04-15 05:43:36] Thanks for the tip. While I wait, I want to give mad props for cutting off the ASICs. This was a giant leap towards decentralisation. [2020-04-15 05:45:57] → msvb-mob joined (~msvb-lab@monero/hardware/michael) [2020-04-15 05:53:12] * kic0 → kico [2020-04-15 05:58:14] → Febo joined (59d411cd@89-212-17-205.static.t-2.net) [2020-04-15 06:15:51] → woodser joined (woodser@gateway/vpn/protonvpn/woodser) [2020-04-15 06:24:01] ⇐ favioflamingo1 quit (~Thunderbi@2405:6580:3c0:5a00:4a45:20ff:fe37:baa9): Remote host closed the connection [2020-04-15 06:24:05] ⇐ favioflamingo quit (~Thunderbi@2405:6580:3c0:5a00:4a45:20ff:fe37:baa9): Remote host closed the connection [2020-04-15 06:26:35] ⇐ Blackwolfsa4 quit (~Blackwolf@195.159.29.126): Quit: Ping timeout (120 seconds) [2020-04-15 06:26:38] ⇐ rdymac_ quit (~britoair@90.162.99.55): Read error: Connection reset by peer [2020-04-15 06:26:55] → rdymac joined (~britoair@90.162.99.55) [2020-04-15 06:26:59] → Blackwolfsa4 joined (~Blackwolf@195.159.29.126) [2020-04-15 06:27:19] * rdymac → Guest72 [2020-04-15 06:27:37] ⇐ TheoStorm quit (~TheoStorm@host-p8vu8h.cbn1.zeelandnet.nl): Quit: Leaving [2020-04-15 06:28:47] ⇐ redfish quit (~redfish@2a00-1dc0-2925--1e.static.as43289.net): Ping timeout: 246 seconds [2020-04-15 06:29:08] A speed boost would be nice. Whether it's worth churn that can introduce bugs depends on how much is gained. [2020-04-15 06:29:32] Last I tried, precompiled headers did not gain much, the patch end up rejected. Maybe your method might gain more though. [2020-04-15 06:30:02] Someone else started working on moving some serialization away from headers. Can't recall the nick now though. [2020-04-15 06:30:23] ccache works fine with monero, I use it all the time. [2020-04-15 06:30:59] I did not know CLANG had a compile time profiler. Sounds nifty. [2020-04-15 06:31:43] ⇐ KeiraT quit (~k4ra@gateway/tor-sasl/k4ra): Ping timeout: 240 seconds [2020-04-15 06:35:38] → KeiraT joined (~k4ra@gateway/tor-sasl/k4ra) [2020-04-15 06:41:03] ⇐ KeiraT quit (~k4ra@gateway/tor-sasl/k4ra): Ping timeout: 240 seconds [2020-04-15 06:41:29] ⇐ spoke0 quit (~spoke0@149.3.71.157): Remote host closed the connection [2020-04-15 06:41:55] → spoke0 joined (~spoke0@149.3.71.157) [2020-04-15 06:42:06] not sure if chromium is the only tool that can open the output :/ https://aras-p.info/blog/2019/01/16/time-trace-timeline-flame-chart-profiler-for-Clang/ [2020-04-15 06:43:17] → redfish joined (~redfish@2a00-1dc0-2925--1e.static.as43289.net) [2020-04-15 07:01:01] → KeiraT joined (~k4ra@gateway/tor-sasl/k4ra) [2020-04-15 07:12:23] moneromooo: What would you propose doing now? It might be faster to just start moving the serialization away, since I have a lot of free time until the mid-May. I don't know how I would find the contributor with over 2.200 forks currently. [2020-04-15 07:14:06] But I see, that there is interest. I think I'll just start then :) [2020-04-15 07:22:30] How about doing something else, since nobody's doing precmpiled headers AFAIK ? :) [2020-04-15 07:23:08] IIRC when I tried it, the reason why it did not gain much at all was we had several directories, and the precompiled thing was being done one per directory. [2020-04-15 07:24:05] Anyone remembers who was asking about changing serialization ? IIRC we talked about the speedup vs churn pros/cons too :) [2020-04-15 07:24:17] This is also cool. But when was the last time, that you heard from this person, who offered moving the serialization away from headers? [2020-04-15 07:24:56] Very roughly, I'd say... a couple months maybe ? [2020-04-15 07:25:11] because I can work full time on this now, until mid-May, meaning I could quickly bring it into life. Then just an hour or two after work. [2020-04-15 07:27:19] Well, your pick really. [2020-04-15 07:27:53] I think I'll give it a shot. [2020-04-15 07:28:44] Historically, I've been pushing against churn (ie, rewriting stuff for no good reason (aka some code standard says so)). I'm trying to do it less now :) [2020-04-15 07:29:06] So build times are a good reason, if the speedup is substantial. I'm curious to see how much you get. [2020-04-15 07:29:51] The point here is obviously not to create any differences in functionality. [2020-04-15 07:30:32] but I know from my experience, that developers usually are careless, and need a janitor :) So if you assume no churn, the mess never gets cleaned up. [2020-04-15 07:32:46] was it woodser ? [2020-04-15 07:36:19] I don't think so. [2020-04-15 07:36:58] ⇐ synaps3 quit (~synaps3@unaffiliated/synaps3): Ping timeout: 256 seconds [2020-04-15 07:53:49] ⇐ coppa quit (~Thunderbi@91-164-241-132.subs.proxad.net): Quit: coppa [2020-04-15 08:18:08] A small finding: [2020-04-15 08:18:10] enjo@tp:~/devel/github/monero-build$ make test [2020-04-15 08:18:11] Running tests... [2020-04-15 08:18:11] Test project /home/enjo/devel/github/monero-build [2020-04-15 08:18:12] No tests were found!!! [2020-04-15 08:20:23] make release-test [2020-04-15 08:21:40] To run one particular tests, make -C $BUILDDIR/subtree test [2020-04-15 08:21:54] eg: make -C build/release/foo/tests/unit_tests test [2020-04-15 08:23:38] make release-test [2020-04-15 08:23:39] make: *** No rule to make target 'release-test'. Stop. [2020-04-15 08:23:48] building out of source. [2020-04-15 08:24:48] You messed up your Makefile most likely. Check the README.md file. [2020-04-15 08:24:53] See git diff [2020-04-15 08:28:09] ⇐ smooth quit (~ubuntu@ec2-54-201-223-245.us-west-2.compute.amazonaws.com): Ping timeout: 272 seconds [2020-04-15 08:43:52] Ah ok. This can't be executed from an out of source directory. [2020-04-15 08:45:29] It can if you know what you're doing. You hinted you know cmake, so look at the contents of Makefile. [2020-04-15 08:46:33] it's been a couple of months since i last tried the tests but iirc it just worked when i did `cmake ..` instead of trying the makefile [2020-04-15 08:48:05] Let's say, that I will address this at a later time. For now I will be happy if the tests execute and keep me from screwing things up. [2020-04-15 08:48:11] → favioflamingo1 joined (~Thunderbi@2405:6580:3c0:5a00:4a45:20ff:fe37:baa9) [2020-04-15 08:48:12] → favioflamingo joined (~Thunderbi@2405:6580:3c0:5a00:4a45:20ff:fe37:baa9) [2020-04-15 08:49:28] ⇐ atomicman quit (~bbhgf@24-156-78-180.flagcmtk03.res.dyn.suddenlink.net): Quit: hehaw [2020-04-15 08:57:38] → atomicman joined (~bbhgf@24-156-78-180.flagcmtk03.res.dyn.suddenlink.net) [2020-04-15 08:59:42] * M5M400_ → M5M400 [2020-04-15 09:18:25] ⇐ CommonDeer quit (~CommonDee@14-201-4-66.static.tpgi.com.au): Read error: Connection reset by peer [2020-04-15 09:18:51] → CommonDeer joined (~CommonDee@14-201-4-66.static.tpgi.com.au) [2020-04-15 09:22:42] ⇐ atomicman quit (~bbhgf@24-156-78-180.flagcmtk03.res.dyn.suddenlink.net): Remote host closed the connection [2020-04-15 09:41:48] → atomicman joined (~bbhgf@24-156-78-180.flagcmtk03.res.dyn.suddenlink.net) [2020-04-15 09:45:36] ⇐ woodser quit (woodser@gateway/vpn/protonvpn/woodser): Ping timeout: 256 seconds [2020-04-15 09:54:54] → woodser joined (~woodser@cam4-17.as22384.net) [2020-04-15 10:05:56] → nonie joined (~irc@p200300C20F131E010000000000000001.dip0.t-ipconnect.de) [2020-04-15 10:05:56] ⇐ nonie quit (~irc@p200300C20F131E010000000000000001.dip0.t-ipconnect.de): Client Quit [2020-04-15 10:15:54] ⇐ favioflamingo quit (~Thunderbi@2405:6580:3c0:5a00:4a45:20ff:fe37:baa9): Ping timeout: 256 seconds [2020-04-15 10:17:02] ⇐ favioflamingo1 quit (~Thunderbi@2405:6580:3c0:5a00:4a45:20ff:fe37:baa9): Ping timeout: 256 seconds [2020-04-15 10:39:27] moneromooo: For how long do the core_tests normally run? I feel like they hung since like an hour. [2020-04-15 10:42:30] Sounds plausible. If you run them often, there's a (closed) patch from me on github that speeds them up a lot. [2020-04-15 10:43:00] Takes ~ 10 min with the patch or so. [2020-04-15 10:44:40] Thank you. [2020-04-15 10:45:10] I will run them for the night then. [2020-04-15 10:46:53] Could you give me a hint, which tests cover the class tx_source_entry? I think I will start from this one as a sample small enough. [2020-04-15 10:47:11] Transaction tests. [2020-04-15 10:47:19] Which is going to be most of them. [2020-04-15 10:51:37] Great. Thanks! [2020-04-15 11:03:00] → TheoStorm joined (~TheoStorm@host-p8vu8h.cbn1.zeelandnet.nl) [2020-04-15 11:17:35] ⇐ DeanWeen quit (~dean@gateway/tor-sasl/deanguss): Remote host closed the connection [2020-04-15 11:18:06] → DeanWeen joined (~dean@gateway/tor-sasl/deanguss) [2020-04-15 12:00:29] ⇐ TheoStorm quit (~TheoStorm@host-p8vu8h.cbn1.zeelandnet.nl): Quit: Leaving [2020-04-15 12:01:15] ⇐ Insight quit (sid415968@gateway/web/irccloud.com/x-bbahcfjkmlxlfhtn): [2020-04-15 12:03:43] ⇐ maxwilliamson quit (~maxwillia@gateway/tor-sasl/maxwilliamson): Ping timeout: 240 seconds [2020-04-15 12:04:24] → maxwilliamson joined (~maxwillia@gateway/tor-sasl/maxwilliamson) [2020-04-15 12:06:23] → wizardsmoke joined (uid209547@gateway/web/irccloud.com/x-dupbbxnakvgedurx) [2020-04-15 12:08:20] → TheoStorm joined (~TheoStorm@host-p8vu8h.cbn1.zeelandnet.nl) [2020-04-15 12:14:15] mjxmr moneromooo selsta : I have a github issue related to serialization, the technique woud reduce compile times as things can be split into a cpp [2020-04-15 12:14:56] https://github.com/monero-project/monero/issues/6406 [2020-04-15 12:15:00] this one? [2020-04-15 12:15:00] other things can be done to reduce compile times, but doing so without re-organizing large parts of the code at once coudl be tricky [2020-04-15 12:15:29] yes [2020-04-15 12:16:08] someone could reduce it further if the macro stuff was a bit much, but the general idea of splitting header/cpp is somewhat straightforward [2020-04-15 12:16:50] → asdc_ccc joined (~asdc_cccc@95-178-173-249.dsl.optinet.hr) [2020-04-15 12:17:04] the biggest compilation issues are the epee server stuff, and its going to be tricky to refactor that without it being a full rewrite (it may have to be to achieve the goa) [2020-04-15 12:17:39] I started some small stuff with hex encoding, trying to remove boost::lexical_cast from being included so often, its fairly heavy to pull in the inculdes [2020-04-15 12:18:27] gcc and clang both have a flag for reporting internal time spent compiling which is useful too [2020-04-15 12:18:55] the flag gives breakdowns on much time spent doing macro expansion, etc., so you can get an interna look at time spent [2020-04-15 12:20:28] ah found the proposal was in the other tab [2020-04-15 12:20:37] → sech11 joined (~sech1@31-208-56-185.cust.bredband2.com) [2020-04-15 12:22:05] yeah moving them into a cpp should help, but the bulk is _probably_ epee, but its hard to profile for certain due to the ways things are compiled [2020-04-15 12:23:47] the boost serialization stuff was separate than what I was working on, and has nearly the same problem+solution [2020-04-15 12:24:06] ⇐ sech1 quit (~sech1@31-208-56-185.cust.bredband2.com): Ping timeout: 265 seconds [2020-04-15 12:29:21] → objectorange_ joined (~objectora@172.58.19.43) [2020-04-15 12:32:14] ⇐ objectorange quit (~objectora@172.58.75.248): Ping timeout: 240 seconds [2020-04-15 12:50:53] → hv-bridge joined (~hv-bridge@web.hashvault.pro) [2020-04-15 12:52:02] ⇐ dethos quit (~dethos@213.190.223.7): Ping timeout: 256 seconds [2020-04-15 12:53:05] ⇐ derpy_bridge quit (~derpy_bri@92.223.89.201): Remote host closed the connection [2020-04-15 12:53:51] → derpy_bridge joined (~derpy_bri@92.223.89.201) [2020-04-15 13:08:30] → KeiraT- joined (~k4ra@gateway/tor-sasl/k4ra) [2020-04-15 13:11:23] ⇐ KeiraT quit (~k4ra@gateway/tor-sasl/k4ra): Ping timeout: 240 seconds [2020-04-15 13:11:26] * KeiraT- → KeiraT [2020-04-15 13:14:19] vtnerd: Shall we coordinate our efforts? My concrete proposal is to completely wrap the boost::archive. Putting the serialization templates can't be achieved, as it turns out, but there's no problem in wrapping the boost:archive. [2020-04-15 13:15:00] → dethos joined (~dethos@213.190.223.7) [2020-04-15 13:15:20] It will be however hard to prove improvements, since even if I wrap the b::archive in one header, it is still included by some other headers, yielding no improvement, until they are sorted out. [2020-04-15 13:15:47] In other words, the project needs *months* of maintenance, just be sure, that you realize it. [2020-04-15 13:16:33] We have to do it one by header type by one. [2020-04-15 13:19:21] EDIT: "Putting the serialization templates INTO .CPP FILES can't be achieved, as it turns out" [2020-04-15 13:40:33] → woodser_ joined (~woodser@cam4-17.as22384.net) [2020-04-15 13:42:49] ⇐ woodser quit (~woodser@cam4-17.as22384.net): Ping timeout: 264 seconds [2020-04-15 13:46:35] ⇐ asdc_ccc quit (~asdc_cccc@95-178-173-249.dsl.optinet.hr): Quit: Leaving [2020-04-15 14:13:24] ⇐ SerHack quit (~SerHack@162.210.173.127): Remote host closed the connection [2020-04-15 14:15:23] → SerHack joined (~SerHack@162.210.173.127) [2020-04-15 14:15:40] → coppa joined (~Thunderbi@2a01:e0a:13c:6110:55e8:2142:245d:8808) [2020-04-15 14:25:31] vtnerd: Wrapping the serialization will make it easier to integrate different serialization techniques later on.