[Patch] "Varlen" Encoding to reduce PPU file size by 10%
Original Reporter info from Mantis: CuriousKit @CuriousKit
-
Reporter name: J. Gareth Moreton
Original Reporter info from Mantis: CuriousKit @CuriousKit
- Reporter name: J. Gareth Moreton
Description:
This is a new facility in the compiler that serves to reduce the size of PPU files by approximately 10% with only a very small performance hit. It seeks to reduce the number of bytes stored for size, count and length fields etc. when they normally store small values.
To achieve this, it uses a customised encoding system named "Varlen" (see http://wiki.freepascal.org/Varlen_Encoding for a technical description). Instead of using a full 4 bytes or 8 bytes for storing values that are frequently less than, say, 100 (but which much support the full range), Varlen Encoding only stores the number in 1 byte for such small values.
Steps to reproduce:
Apply patches and confirm correct operation and reduced file size of PPU files.
"varlen_" patches are independent, although all depend on "varlen_base.patch", which contains the encoding implementation (and increments the PPU version by 1).
"misc_ppu_optimisations.patch" contains optimisations that aren't Varlen-related, like compressing concurrent Boolean fields.
"misc_util_ppudump.patch" updates the ppudump utility to correctly decode Varlen-encoded fields (requires "varlen_base.patch")
Additional information:
I don't consider Varlen to be a compression technique in the same vein as ZIP - it behaves more like UTF-8. For the largest of values, Varlen requires 9 bytes of storage, so is not suitable for things like hashes.
Additionally, some minor savings were made by compressing concurrent Boolean fields into a single byte. If anything, this is faster to read as well because only a single byte is read from the stream and then an "and" operation is performed on each relevant bit.
Though high-end systems don't have major issues with storage space, for smaller systems, every little helps.
Mantis conversion info:
- Mantis ID: 35299
- OS: Microsoft Windows
- OS Build: 10 Professional
- Build: r41802
- Platform: Cross-platform
- Version: 3.3.1
- Target version: 3.3.1