chaokunyang

I get your concern — for one or two languages, skipping an IDL can work well and keeps things simple.

But once you’re dealing with three or more languages, I agree an IDL becomes valuable as a single source of truth. That’s work we’ve started: adding optional IDL support so teams can generate data structures in each language from one shared definition.

This is our first release for Fory Rust. The Java and Python bindings has been used widely. You can see from https://fory.apache.org/user

You don’t need to hand‑write serializer code. In typed languages you just define your class or struct as usual; in dynamic languages you can use type hints.

When running in compatible mode, Fory automatically derives a compact schema from those definitions at runtime time and sends it along to peers for the first time serialization. That way, both sides know the structure without needing a separate schema file.

The idea is to make cross‑language exchange work out‑of‑the‑box, while still allowing teams to add an explicit IDL later if they want a single source of truth.

Probably not for everyone. The current limit of 4096 types could be expanded if there’s a real need — it’s not a hard technical barrier.

I’m curious though: what’s an example scenario you’ve seen that requires so many distinct types? I haven’t personally come across a case with 4,096+ protocol messages defined.

That’s a fair point — with more languages in the mix, having a formal schema can definitely help prevent drift.

The way I think about it is: • Single‑language projects often work best without an IDL — it keeps things simple and avoids extra steps. • Two languages – both IDL and no‑IDL approaches can work, depending on the team’s habits. • Three or more – an IDL can be really useful as a single source of truth and to avoid manually writing struct definitions in every language.

For Apache Fory, my plan is to add optional IDL support, so teams who want that “single truth” can generate definitions automatically, and others can continue with language‑first development. My hope is to give people flexibility to choose what fits their situation best.

No, it's not needed. We plan to remove Google Guava from the Fory Java dependency. Our philosophy is that the core should have as few dependencies as possible for maintainability and minimal footprint.

Apache Fory is also a drop-in replacement for pickle/cloudpickle, you can use it to serialize code object such as local function/Classes too.

https://github.com/apache/fory/tree/main/python#serialize-lo...

When serializing code objects, pyfory is 3× higher compression ratio compared cloudpickle

And pyfory also provide extra security audit capability to avoid maliciously deserialization data attack.

Apache Fory is a serialization framework that works across languages without schema files. We just released the Rust implementation (v0.13) with some features we haven't seen elsewhere:

  1. Serialize Box/Rc/Arc<dyn Trait> and preserve polymorphism on deserialization
  2. Automatic circular reference handling (parent-child trees, graphs)
  3. Reference identity preservation (Rc/Arc pointer equality maintained)
  4. Cross-language compatibility (Rust ↔ Python/Java) with no IDL
  5. Schema evolution without breaking changes
  6. 10-20x faster serialization than JSON/Protobuf

Apache Fory is a blazingly-fast multi-language serialization framework powered by JIT (just-in-time compilation) and zero-copy, providing up to 170x performance and ultimate ease of use.

The Apache Fury team is now officially announcing that Apache Fury has been renamed to Apache Fory, effective immediately. This transition impacts all project components including code repositories, package names, documentation, and communication channels.

Available since https://github.com/apache/fury/releases/tag/v0.10.0

See https://github.com/apache/fury for details about Apache Fury serialization framework

Apache Fury is a blazingly-fast serialization framework for jdk. Apache Fury use automatic codegen for object graph serialization at graalvm native image build time. It eliminate all reflection and introspection at runtime. In this way and its optimized serialization protocol, It runs up to 50x than graalvm jdk serialization. And user don't have to specify reflection config json file, no annotation are needed. It's easy to use and minimize user mind overhead.

The code are located at:

1. https://github.com/apache/fury/blob/main/java/fury-core/src/... 2. https://github.com/apache/fury/blob/main/java/fury-core/src/...

JSON/Protobuf used a KV layout when serialization, it will write field names/types multiple times for multiple objects of same type. And the sparse layout is not friendly for CPU cache and compression.

We proposed a scoped meta packing share mode in Apache Fury 0.6.0 which can improves performance and space greatly.

With meta share, we can write field name&type meta of a struct only once for multiple objects of same type, which will save space and improve performance comparedto protobuf. And we can also encode the meta into binary in advance, and use one memory copy to write it which will be much faster.

In our test, for a list of numeric struct, Fury is 6x faster and 1/2 payload smaller than protobuf.

Pretty cool, multiple level IS is great. Does KQIR support predicate pushdown?

We've already support such dict encoding, we let users register class with an id. And write class by id, id will be encoded as a varint, which uses only 1~5 bytes. But not every users like the registration. Meta string encoding here is for cases where not mapping is available

Meta string is used only for ascii chars, so every char is in range of a byte. We can just random access them. But your reminder is good, I just find out that we didn't check the passed string are ascii string, although our case always pass ascii string

Glad to see H2 database engine benifits from this. I use H2 too.

Yes, if we can. The users of Fury may be able to do this. Fury may only provide such an interface to users to let them pass such an dict/zstd/huffman or something like

Yes, we only use this encoding when it brings benifits. `wire format` is a good term to describe such cases, which the context are constrained to current message only. So the data for compression are small, and many statistical methods won't work since it's a `wire format`. The stats itself must be send to peers, which will have bigger space and cpu cost

The thing is that we can't store a precomputed dictionary. Fury is just a serialization framework, we don't know which data will be serialized

Hi, it's not “sent only once” . It's millions of “sent only once” .

The thing here are that all those RPC are stateless, so the context for compression are the one message itself. i.e. compress object like `Point(1,2)` only without priori knowledge. Users may use train static zstd. But Fury can't, Fury doesn't know which data will be serialized.

And meta string encoding are not statistical, it won't beats zstd. It's just for cases zstd is not suitable.

Sadly, we don't have such a training corpus of representative data. Fury is just a serialization framework, we can't assume any string distribution. I thought about scan the code of apache ofbiz, and use the domain objects in this project as the default corpus to carry a static huffman/zstd. But ofbiz may not be representative still. For your second suggestion, I can't agree more. Actually Fury has already implemented this, we call it meta share mode. Fury will write such meta only once on a channel. And resend such meta if the channel got disconnected. But this is not easy to use and impose overhead to users. So we proposed meta encoding here. Anyway, yours suggestions are very insightful. Thanks very much

Dictionary encoding is already used in fury. We will encode same string as an varint when it's seen later. The thing here is that many cases the string don't have repeat. Imagine you send `record Point(into x, int y)` in an rpc. You have one 'Point' to send. No dictionary encoding will be used in such cases

This is interesting, thanks for sharing this. I will take a deep look at it.

We have such options in Fury too. We support register classes. In such cases, such string will be encoded with a varint id. It's kind of dict encoding. But not all users like to register classes. Meta string will try to reduce space cost when such dict encoding are not available.

Your suggestions are great. I should clarify why dict encoding can't be used to recude cost.

As for performance, it won't be an issue. The string for meta encoding are limited, we can cache the encoded result. So it's not in critical path

We can use `|` instead, `|` is not used in path. Currently we have only 2 chars can be extended. I haven't decided what to inlucde

Good suggestion, I was planing it as an optional option. We can crawl some github repos, and extract most types, extract their class names, paths, fields names, and use such data as the corpus

The thing is that we don't the frequency about every char happens