HNHacker News
TopNewBestAskShowJobs

status_quo69

271 karma · joined November 29, 2016

submissionscomments
status_quo69··on How I Start: Rust
So the answer to both of those things is that I basically hammered out a bit of code that looks sort of like what I have but it's exactly. It should have been a 503 error instead.

In reality, the main thought is that the map_err in the controller should take some error and match against it for a known set of Error enums to translate to an enum that we want to get back out to the user.

> Also, I take it this code returns a body of { status: ..., message: ... } with a status of HTTP 500? So the enum variant "Unavailable" is just for programming syntax and unwraps around the Json<T> inner body when it's called by Rocket?

Yep! Exactly. It's a bit wonky but I thought it would be helpful to show off, especially in a world where JSON bodies with a shape of { status: "error" ... } are so common. In reality you can shove whatever you want in there and use serde to flatten the result. Off the top of my head, you could in theory do something like

    pub struct UserApiErrorMessage {
        status: String,
        message: String,
        #[serde(flatten)]
        _meta: HashMap<...>
    }
Where _meta holds some additional information you might want to dump to the user. In practice, I'm using this error enum approach to construct JSONAPI responses with appropriate status codes and the appropriate structure.

> Finally, I see you are making your diesel calls inside the controller.

So what I've found is that generally you actually don't want to embed your diesel get_result(s) calls inside your impls for your struct, since you're loading the entire set up into memory at once and you lose the ability to cut down. I've leveraged the `into_boxed` method of the query builder pretty heavily to allow for building up queries on the fly, which allows me to abstract the common bits into the impl. Code shows better than words so here we go:

    struct User { .. }

    type WithUsername<'a> = Eq<users::username, String>;

    impl User {
        pub fn read_all<'a>() -> BoxedQuery<'a, Pg> {
            users::table.order(users::created_at).into_boxed()
        }
 
        fn with_username(username: &String) -> WithUsername {
            users::username.eq(username)
        }
    }


    // In some method somewhere. Note: I don't know that this actually works out of the box because I didn't compile it
    User::read_all().filter(User::with_username(&username)).paginate(1).per_page(25).get_results::<User>()?
So I've cooked up a sort-of ORM for this, but I've abstracted away that underlying schema.rs file that's a huge part of diesel, since I think (but haven't proven) that leaking that file out into the rest of the codebase makes things tougher in the long run.

As to your point about the controller, I'd advise against it but hacked something together for the comment. In my project, I actually call out to simple services (think shitty service oriented architecture) that query the database and return a Result. What's interesting (or not) here is that my services actually call the `Into::into` of the database model, so my controller never even sees that they exist. Again, some code:

    // Actually in some other file
    mod service {
        mod users {
            pub fn get_user_page(conn: &PgConnection) -> Result<Vec<UserApiResponse>> {
                let users = User:read_all().paginate(1).per_page(25).get_results::<User>()?;
                users.iter().map(Into::into).collect::<Vec<_>>()
            }
        }
    }

    #[get("/users")]
    fn index(connection: &PgConnection) -> Result<Json<Vec<UserApiResponse>>> {
        service::users::get_user_page(&*connection).map(Json).map_err(...)
    }
But now we come to what I think is the most pertinent question: why does this exist? Testing. Spinning up a DB in CI is quite easy but since rust runs all testing in parallel (yay fearless concurrency!), we can get into trouble when we delete some records from the database but we haven't set up our tests to handle that. I ran into some really stupid problems I caused for myself in assuming that somehow it would be like rspec. If you can basically abstract away the database access part into a dumb hashmap, you get a store per test that won't experience much contention (at least that's the hope). I'm still working on that last bit so my thoughts on it are far and away from complete.

EDIT: Final note on diesel: there is no into_boxed for inserts and into_boxed for updates is dark magic that I can never get working correctly, so I'd avoid being too clever with these and focus on Select/Delete instead

status_quo69··on How I Start: Rust
The best way to learn rust depends on your learning style, the Book (https://doc.rust-lang.org/book/index.html) is fantastic even as a reference. To learn rocket I'd start with the official tutorial here: https://rocket.rs/v0.4/guide/getting-started/#getting-starte...

It might sound canned, but the rust community generally has fantastic documentation and the few edge cases you'll hit are generally well-known in the github issues with well-known causes. My advice would be that if you feel like the compiler is giving you too much grief, take a break for a day or two and then come back to it, you might find that what was giving you trouble isn't so bad anymore. The compiler is very helpful with error messages but because of the litany of information that it gives you it can become a bit of a brain overload, especially if you hit a particularly nasty diesel type error (compile time SQL is no joke!)

It's also good to know in advance that the rust standard lib is great, but fairly bare bones if you want things like JSON support. Check out https://www.arewewebyet.org/ for the tools that can help you out!

EDIT: The only other thing that I just thought of is that authz/authn doesn't have a good story in rust just yet, so I'd rely on an external service of some kind to handle that for you unless you want to do basic signed session cookies.

status_quo69··on How I Start: Rust
Yep! Serde allows us to cheat a bit since you can call for it to skip certain fields (such as password hashes), but my preference is for the field to not exist at all in your serialization structs. Less of a chance to forget that #[serde(skip)] flag
status_quo69··on How I Start: Rust
Sure! Let's start with two baseline diesel structs:

    #[derive(Serialize, Deserialize, Identifiable, Queryable)]
    #[serde(rename_all = "camelCase")]
    struct User {
        pub id: Uuid,
        pub username: String,
        pub created_at: DateTime<Utc>,
        pub updated_at: DateTime<Utc>,
    }


    #[derive(Serialize, Deserialize, Identifiable, Queryable, Associations)]
    #[belongs_to(User)]
    #[serde(rename_all = "camelCase")]
    struct Business {
        pub id: Uuid,
        pub name: String,
        pub user_id
    }
And their respective endpoints:

    #[get("/users)]
    fn user_index(connection: &PgConnection) -> Result<Json<Vec<User>>, Status> {
        // Going to pretend like we have some helper methods to help us out here
        User::read_all().get_results::<User>(&*connection).map_err(|_| Status::Unavailable)?
    }

    #[get("/companies")]
    fn company_index(connection: &PgConnection) -> Result<Json<Vec<User>>, Status> {
        // Going to pretend like we have some helper methods to help us out here
        Company::read_all().get_results::<Company>(&*connection).map_err(|_| Status::Unavailable)?
    }
This is all well and good until maybe you have a UserPreferences model, which may or may not need its own route. Unlike other languages, you can't easily add attributes to structs. So if you made the choice to embed your UserPreferences into your User api response, you'd have to either: craft a json object from the json! macro, or alter all of your models anyways to introduce the new structure. Now your project/response structure would be:

    #[derive(Serialize, Deserialize)]
    struct UserApiResponse {
        id: Uuid,
        username: String.

        preferences: Option<UserPreferencesApiResponse>,
        company_id: Option<Uuid>
    }

    impl From<User> for UserApiResponse {
        fn from(user: User) -> Self {
            Self {
                id: user.id,
                username: user.username,
                preferences: None,
                company_id: None,
            }
        }
    }

    impl UserApiResponse {
        pub fn with_preferences<P: Into<UserApiPreferences>>(&mut self, preferences: P) {
            self.preferences = Some(preferences.into());
        }

        pub fn with_company<C: Into<CompanyApiResponse>>(&mut self, company: C) {
            self.company_id = company.id;
        }
    }

    #[get("/users)]
    fn user_index(connection: &PgConnection) -> Result<Json<Vec<User>>, Status> {
        let preferences = Preferences::read_all().get_results(&*connection).map_err(|_| Status::Unavailable)?;

        User::read_all().get_results::<User>(&*connection).map(|user| {
            let mut user: UserApiResponse = user.into();
            user.with_preferences(preferences.first().unwrap());
        }).map_err(|_| Status::Unavailable)?
    }
The code doesn't work out of the box but I hope that it conveys the idea that I'm trying to get across. This is sort of the same idea as Marshmallow in python or ActiveModelSerializers for Rails, but the problem is more pointed in rust because at least in python/ruby you can just shove on values as you need them (debatable if this is a good thing). The other thing that we've gained is consistency. Any change to an underlying serialization model is automatically reflected in any endpoint that may utilize it (also debatable if this is a good thing).

To really drive the point home, consider what would need to happen if we serialized out the database models directly from a REST standpoint. The steps would be:

    1. Fetch the user
    2. Fetch the preferences by filtering on user id (requires a query parameter)
    3. Fetch the company by filtering on user id (requires a query parameter)
In this new approach, we cut only one response, but if we were using GraphQL/JsonAPI we can cut this down even more.

If we stuck with our base schema we'd be locked into always modifying our database schema models, which IMO ties too much to the data access layer, or having separate fetches per request which is non ideal for a 250ms response time per request. In this world, we can go a step further without ever altering our DB models, which allows us to reason about them in a more dumb/CRUD way.

status_quo69··on How I Start: Rust
One thing I wish was covered in these is how to go past 'hello world', especially things like error responses and serialization models. It took me a while before I got to the point where I think I can be productive in both querying the data from the front-end and getting back the shape that I wanted. Couple of lessons learned:

- Start with separating your API models and DB models out early, you'll be thankful later. You can also help yourself out by implementing From/Into for your database models to your API models. Obvious lesson from CQRS but since rust doesn't allow easily for ad-hoc structs this is a must, especially since there are a lot of examples out there that put a Serialize/Deserialize onto the database object.

- If you have deeply nested objects, look into JSON:API or Juniper. There are a couple of crates that implement the jsonapi spec and I'm hacking on one of them to make it a bit more usable for my personal pet project, but Juniper will make your life easier especially if you're starting with a React frontend.

- Understand that there's still a tracking issue for a few items such as multipart data, although there's a third party crate that integrates the multipart crate into rocket. However if you need an upload you could probably get away with just returning a signed S3 endpoint to the user, which is what I ended up doing

- If integrating with redis or some other database that's not application critical, there is no (as far as I'm aware) graceful degredation that comes out of the box with rocket_contrib, unless there was some config value I was missing. This is usually fine, since you can easily declare an ad-hoc fairing that connects to redis and falls back to an Option::None in the case where it couldn't connect.

And finally, if you want to support different error status codes yourself based off of mappings from internal errors (such as 409 conflict when inserting the same data), your best bet is an enum that's wrapped by a responder derive like so:

    #[derive(Responder, Debug)]
    pub enum UserApiError {
        #[response(status = 500, content_type = "json")]
        Unavailable(Json<UserApiErrorMessage>),
    }

    impl UserApiError {
        fn wrap_body(message: &str) -> Json<UserApiErrorMessage> {
            Json(UserApiErrorMessage {
                status: "error".to_string(),
                message: message.to_string(),
            })
        }

        pub fn unavailable(message: &str) -> Self {
            Self::Unavailable(Self::wrap_body(message))
        }
    }
Which can be called like:

    Err(UserApiError::unavailable("The database is gone!"))
Most of my other lessons learned come from diesel, both are an absolute pleasure to work with and have given me incredible confidence while coding.
status_quo69··on How I Start: Rust
I've used it for 6 months on the async branch and the master branch and master was incredibly stable. Async is less so because it's a large scale effort to try and convert over the entire API with minimal breakage(apart from fairings/middlewares). You can pin to a specific stable nightly if you're needing extra sanity, although CI makes this hard with the nightly docker images. There's a tracking issue for proc_macro_hygine here: https://github.com/rust-lang/rust/issues/54727

Looks like the last feature is almost stable!

status_quo69··on Managed Kubernetes Price Comparison
Load balancer certs via annotations are supported, but they're a bit iffy when pairing with controllers like ambassador, since ambassador expects to own TLS termination (although the ambassador docs do say this is configurable). https://www.digitalocean.com/docs/kubernetes/how-to/configur...
status_quo69··on A Sad Day for Rust
If my comment implied that there was some sort of expectation then I think there was a miscommunication. There's no obligation for anyone to do anything, ever, but that's a pretty useless metric to mark against IMO.

Was this a professional project? Absolutely not. But was this a personal project? I would also say categorically "no", since it wasn't scoped under his personal github profile until recently. I'm very saddened to see something that could have been solely community maintained be revoked instead of there existing a potential mitigation plan in place (for example in the form of a foundation or Org) for situations like this, especially since this has happened twice before and it's sadly caused burnout for the main developer. Again, no obligation to do this but why can't I say "man, that's true that he could follow these actions but that's also shitty" at the same time?

status_quo69··on A Sad Day for Rust
I want to preface my comment with "I agree with you"

However, in this case patches and reproductions were submitted and rejected as "boring". Whether the author really meant to use the word "boring" is neither here nor there, the result was that the fix was ignored, even though that's how open source works!

I've watched from the sidelines with no skin in the game (I use Rocket) and my conclusion from all of this is that the maintainer should have created an actix committee or group of maintainers to help shield from the inevitable bugs and issues that crop up. If the OG author wanted to step back to take a break from all of this, then fine, the rest of the group can continue to churn and work.

The original article that called out the behavior also called out other incredibly popular crates, like reqwests, for their underlying HTTP stack, yet there's not a shitstorm about those libraries. The obvious difference that I can see is the maintainer response to said criticisms.

EDIT: I also want to make clear that I hated the comment about never writing rust again. I'm not condoning that behavior at all

status_quo69··on Eris – A better way to handle, trace, and log errors in Go
Or you can use a boxed error iirc. Matching on the type of error after you do that isn't exactly great but it's doable if all you want to do is bubble the error up.

Places where that approach really sucks include web api routes (unless I haven't figured out a trick yet). For example, rocket defines Responder on Result so long as the ok and error variants both define it, but since a boxed error is erased it doesn't function correctly.

status_quo69··on Ask HN: Other than REST and GraphQL, what do you guys use?
Rest is (simplified) just a way of expressing your routes as resources. If you want to post and get some things that's perfectly fine but you'll probably wind up accidentally fitting into a different paradigm (rpc for example if you start naming your routes getFooBar and newFooBar).

You can also post form data instead of json if you think json is overkill but getting data from the server is probably limited to something like json unless you want to decide on a custom format or your data is truly so simple that you can just return a single text blob.

status_quo69··on A lighter V8
You can't change the inheritance as far as I know after creation without some hacks, but you can change the class that an instance refers to which can kinda sorta achieve the same thing. You can't necessarily add properties to a base class and have all of those reflect immediately unless you use some hackery with class properties.

Example: https://github.com/dabeaz/python-cookbook/blob/master/src/8/...

status_quo69··on How to learn D3.js
No article, but I've done it. You basically have to take on a lot of the creation of the SVG element tags yourself, since having both D3 and React manage the DOM is a bit awkward. However, a lot of D3 functions already give back most of the properties that you need to construct the DOM anyways. For example, the classic sunburst chart (example here: https://observablehq.com/@d3/zoomable-sunburst) can be constructed from the section of code that contructs and partitions the hierarchy:

    partition = data => {
      const root = d3.hierarchy(data)
         .sum(d => d.value)
          .sort((a, b) => b.value - a.value);
      return d3.partition()
          .size([2 * Math.PI, root.height + 1])
        (root);
    }
Some downsides to this:

* Zooming is non-trivial, you have to recreate it in React proper

* React has to re-render entire hierarchies or massive lists depending on the traversal method that you choose to use for hierarchical datasets (topological or depth first) and React chokes on these pretty easily since they tend to be wide and flat

* Quickly changing data sets can't really benefit from memoization or React.memo, since the layouts themselves will constantly change. It makes things a bit awkward when mixing with memoizable data.

However, once we got the groundwork down, we never really had to touch it again. It was much easier to maintain than the pure d3 implementation using useEffect or componentDidMount, since we didn't have to deal with any sort of DOM state ourselves any more.

status_quo69··on Using TypeScript with React
I won't argue that getting strong code in TS is easy but I definitely don't think it's as hard as you're making it out to be. That being said, it's more of a limitation of the underlying execution context in my opinion, and you have to realize that it's not just jS that's out to get you but the whole ecosystem. But that's the expectation in all languages, even in Rust. In production optimizations all bets are off and you're on your own if your app dies.

For example of what I mean by the platform being out to get you, the second parameter to JSON.parse is a recovery function. So it'll happily parse anything and return anything. Again, should be "unknown" but it was typed before unknown was a thing.

    JSON.parse('{ "fooDate": 123456789 }', (key, value) => !key.includes('Date') ? value : new Date(value));

    JSON.parse('123456789', (key, value) => new Date(value))
I actually just learned about this today, since I was doing some digging. Horrifying stuff for a typed language to get around. FWIW, we've made it an explicit lint error to use the `any` type, and enabled --noImplicitAny and --noImplicitReturns. That's been pretty solid for our codebase and because I/O is limited to a few edges, we can call out those edges for type guards.
status_quo69··on Using TypeScript with React
FYI you can use 'x' as const nowadays.
status_quo69··on Using TypeScript with React
> It feels like people are lulling themselves into a false sense of security by making increasingly complex self-consistence schemes via type utilities, but they just shrug when I point out that the foundation of the scheme is still unsound.

I don't agree with this premise. Assembly is an untyped language but you can build things on top of it like Rust or Haskell, or you can write in C and cast everything to a void. I do* agree that validating the types of values is a hard problem in any language and that JSON.parse should be typed as `unknown` these days (although there are also reasons why it should not do that due to casting), but it makes complete sense IMO. JSON.parse could return any valid JSON value. The compiler doesn't know the structure of the string provided beforehand.

status_quo69··on Anyone made the jump from MySQL to PostreSQL? It is worth it?
Sounds like you don't want an ORM but a DB driver like psycopg2 http://initd.org/psycopg/docs/cursor.html#fetch
status_quo69··on PostgreSQL Features to Try
The one thing I wish that NOTIFY had as a feature would be the ability to do NOTIFY [N], or at least NOTIFY 1. I was writing a very very basic job pool previously and had to try to overcome the thundering herd problem but at the end of it couldn't. I know it's a niche case and maybe there is a clever way of dealing with it, but the only way that I could find to avoid re-doing work and/or trying to select jobs that weren't taken was to allow each listener to be notified and lock the row, where each query for a job would be roughly SELECT * FROM jobs WHERE .... FOR UPDATE SKIP LOCKED and then bail out if I couldn't find any jobs. I could have been using NOWAIT instead, it's been a few years. We weren't looking at a huge number of async jobs to occur but needed reliability and auditing. Ultimately I believe that we scrapped this whole idea so it's all moot but it definitely would have made prototyping a bit easier
status_quo69··on Announcing Rust 1.33.0
I do, it's generally pretty nice and there are a lot of well thought-out libraries available. However, figuring out how to interop between a scripting language and an ECS isn't the most trivial thing since rust's type system doesn't always want to be flexible, YMMV. I'm currently using GGEZ for rendering and utility functions, but if you wanted something a bit larger and more opinionated/mature, amethyst is your current best bet. I'm happy to answer any other questions about it! It's been an interesting learning experience so far and I plan to put a lot more down on paper once I have at least a small game running end to end.
status_quo69··on Happy Birthday, Ruby
That's because std::string does not carry any sort of encoding information, std::string is basically a wrapper around bytes (hopefully I'm not misreading this, I'm far from an expert C/C++ programmer). Due to this, python can't make any assumptions about encoding/decoding without the possibility of getting it wrong.

"Note that this class handles bytes independently of the encoding used: If used to handle sequences of multi-byte or variable-length characters (such as UTF-8), all members of this class (such as length or size), as well as its iterators, will still operate in terms of bytes (not actual encoded characters)."

https://stackoverflow.com/questions/1010783/what-encoding-do...

http://www.cplusplus.com/reference/string/string/

status_quo69··on Ask HN: How did you decide what problems to solve in your lifetime?
I used to work with a healthcare company and dealt the 837 file format and FTP batches and wrote a pretty neat-o parser for it, the reason is simple- inertia. And it's not like people aren't trying because there's been a huge push to get REST-like APIs in place, but when you have close to 40 years of software infrastructure in place it's non-trivial to explain to the boss why you need to spend money on something that gets everyone paid either way.

I think it would be truly revolutionary if we could change the format, but what I think would be almost a better step is to write open source parsers that can accurately translate X12 into something sane and keep the interchange the same so that everything can be balanced out like a checkbook. Check out the 835 format for this part as it's supposed to be a data stream of transactions. Too bad the dozens of flavors of X12 are locked behind licensing and thousands of dollars of books for the rules :/

Additional reading on this god-awful data format: https://docs.oracle.com/cd/E19398-01/820-1275/6ncv5s178/inde...

status_quo69··on Parsing JSON is a Minefield
I'm not going to disagree that data formats should undergo a process.... but sometimes they do undergo too much process. Most of the EDI standards, for example, are a product of a huge amount of committee that sells a lot of standards and are used extensively yet are almost impossible to write a parser for.

Edit: obviously this isn't the case, since people have actually parsed it, yet it's a non-trivial thing to do and I have yet to see a solution that isn't horribly specific or a solution that ignores something that might be problematic in non-standard solutions (of which there are a million because of the difficulty involved). A huge amount of domain specificity doesn't help either.

← PreviousPage 3 of 3