TypeScript on the GPU with TypeGPU creator Iwo Plaza

Welcome to Syntax. Today, we're gonna be talking to Evo Plaza about type GPU and all of the amazing things you can do with it. So we're here at JS Nation, myself, CJ. Welcome, CJ. What's up? And, Ivo, we we found him. Just got off stage here, so we figured we would chat about TypeScript.

Thank you so much for having me. Yeah. It's amazing. Long time watcher of the podcast.

Battles. Oh, yeah. He's a c CJ bot in the comments. CJ bot in the comments. Man. Yeah. So let's start off with with type GPU, which is something that came onto my radar recently. I made a video about it. And, can you just give the audience a little bit of

high level what is TypeScript, how it came about? Well, I I feel like we're in this new kind of era on on the on the web. Like, for a while, all we could do was WebGL and and and graphics, and, three JS is is the main, like, library player. But I feel like, now we have a very good opportunity to try different direction, try something new for TypeScript enthusiasts, and that's what TypeGPU is. TypeGPU is a wrapper around Wes GPU, which is an, an upcoming web standard, which is almost supported in every browser.

Linux is lagging, lagging behind, but hopefully next year.

And it gives you very low level access to your GPU while still, having that TypeScript developer experience.

And our main frontier right now is allowing you to write shaders in JavaScript, which normally, what you have to do is you have to write them in a special language called WebGPU shading language Mhmm. Which, might not be super accessible for existing JavaScript developers.

So we wanna just make that easier. Yeah. I I found the shader language to be very tough to pick up, especially for, like, somebody who's used to programming in JavaScript. You're always thinking about, looping in regards to executing this, then this, then this. But with Yeah. Shaders, you're executing tasks on every single pixel at once when you're doing graphics.

And is that still the case with with type GPU and and how you yeah. Man, that is a a a weird way to, like, shift your brain. Yes.

Have you worked a lot with Wes GPU before pnpm embarking on this or graphics programming in general? Yeah. I mean, my first

experience with programming was with c plus plus and OpenGL a while back. Yeah.

And I only got into JS because they didn't they didn't have, c plus plus compilers on my school computer.

Yeah. So now I'm here. Now now I'm writing ESLint JS. Have you ever built a visualizer

for,

what is it? Like Winamp? Yeah. Like a Winamp visualizer. Yes. Oh, I wish. I wish. It would have been amazing. Mostly, like, game game development Cool. With with OpenGL and and and c plus plus. And, I mean, that was that was always amazing, but I feel like JavaScript is a is a really amazing platform and the browser in general for for creating experiences that you can just share very quickly with people. I was always frustrated that, you know, I was building on my local machine, but if I wanted to share it with a friend that has a Mac, I I couldn't do it. I would have to buy a Mac.

And that just wasn't in my budget GitHub on my 0, dollar budget.

So I feel like the web opens that up a Scott, and I'm thankful that I get to use this experience, in at work and working open source full time.

I've been working a little bit with WebGPU before, and type GPU is mainly the result of my frustration with WebGPU.

It is. It JS. Yeah. Because it's super powerful, but it's very low level.

If you wanna communicate with, with WebGPU, it's essentially like you're trying to talk to a server, but the only way to talk to it is to send instructions to it in a different language and, send the binary to it.

You can't send, like, even numeric values, like, you have to package them up as a a float free to your array and send it over. If you wanna do more complex structures, you have to think about padding in memory, how the GPU is gonna interpret that, what the padding rules are. And if you get that wrong, well, you get a wrong pixel, and good luck trying to debug that. So that was our first frontier just making that connection between the shader and JS easier, and we do that through Node like schemas. Yeah. So we we're very inspired by by libraries like ZOD, like ValleyBot, Arkype, where you define the scheme at once, and you get both the type safety and the runtime checks. And we do a similar thing, but, instead of runtime checking and validating the object, we use it to serialize and deserialize plain old JavaScript objects, arrays, and and primitives. And what that allows us to do is you have a single source of truth that generates both the Wiges or shader code and allows you to send and download the arbitrary data from the GPU.

And that was our first frontier. We finished that up about a year ago. And since then, we've been pushing that forward to allowing you to write shaders in JavaScript, where now, thanks to that integration, you can write your shader and refer to something that lives on the CPU.

And then you just have that connection both at runtime and on the type level.

And that was very, very important for us, and I feel like it unlocks a lot of possibilities similarly to how tRPC and and SvelteKit allow you to run your function

on the server without having to worry about Wes APIs and stuff like that. You can now just call a shader function without having to set all of that glue up yourself. Yeah. Can we talk a bit about how that actually works? Because I mean so, like, if you if you look first of all, go check out the docs, the the paper. Really good docs. Really good. But you're writing code that just looks like TypeScript I don't know if I've seen a library like this before where you're writing TypeScript code, but there's this little use GPU directive. Yeah. And at the end of the day, your code actually gets run through a compiler. Like, you wrote a compiler that takes your code that looks like TypeScript and actually turns it into shader code or, like, web, Wes GPU code. Yes. So, yeah, can you talk a bit about how that works? And, so in your talk, great talk, by the way Oh, thanks so much. You had an analogy of, like, front end versus back end. There's typically, like, this translation layer of, like, a tRPC or even, like, SvelteKit with, like, remote functions. And you mentioned that this type GPU library is kind of like that for the CPU and the GPU. It's like Yes. It makes it much easier to write code that communicates between the two. Can you talk a bit about that and how this all works? Yeah. Yeah. Of course.

I mean, the I feel like the serialization and the type safety is a bit easier to to explain and and start from.

The the code structure, I mean, how how you would approach it normally is just a lot of TypeScript generic magic. Mhmm.

But, essentially, if you like, we have a couple of built in primitives that we just export, and we know the types of them. And if you use them in our APIs, we're able to extract the information about what you're trying to use. So it's essentially equivalent to having, like, multiple overloads over a function, and we choose behavior based on that. But the more more complex data structures like structs that define shapes of objects or or arrays, Yeah. They're just basically a bit more complex in terms of the TypeScript types. So we just go over the fields that you defined in the schema, and we save that on the type level so that when you refer to that same struct, we're able to properly change up the API, and react to it. I mean, like, TypeScript is trained complete on the type level.

So you can technically do anything, even run doom, of course.

But, yeah, it it like, as far as TypeScript magic goes between, defining type annotations and Doom, we're still, like, pretty close to here. Not doing super advanced stuff in the on the type in inferenced, on the type inference front. But, yeah, that's essentially how it works. And afterwards, I mean, since you can just access those objects on in JS, if you write your shaders in JS, you also have access to that. And the types naturally naturally flow from your CPU code to your GPU code since you can import stuff, from other JS files, and the types propagate naturally thanks to it being in the same language.

So we don't even have to, like, write our own custom language server because it since it's just TypeScript and it's semantically correct TypeScript, we can leverage existing tooling, and hook into your existing, bundler, thanks to pnpm in the ecosystem.

Thank you to Daniel Node.

We're able to create a single transformation plug in that looks at your JavaScript, and tweaks it a bit so that it works. I can I can talk about it later if, if you guys want? But, we're able to target pretty much every bundler on the planet, thanks to the work of the Andreas team and Unplug In. And if you, you guys are familiar with the with the project. But, yeah, it's it's it's an amazing piece of tech. And if you if you guys are if you guys think that you want to build something, first look at the Anjaya's website. Nice. Yeah. It's it's You're most likely you're, like, you're most likely gonna find it there. It's cool. Yeah. It's it's amazing. But, yeah, we basically look at your JavaScript source code source code. We look for functions that are marked with the use GPU directive, and we attach metadata to it so that at runtime, we can, associate a function with the metadata that we can use for generating the shader code. So the actual build step is quite simple to showcase and to explain, at least, like, visually what the code actually turns into.

You could think that we replaced the JavaScript functions with the shader code, but that's not actually what happens.

To keep it flexible, and to allow the shaders to react to the user's settings or, or your settings as as the developer, or as I showcased in my talk, being able to accept callbacks from the user to inject into your shader Node.

We do have to defer shader code generation to runtime.

You can do that at build time if you use another unplugging gem or on JS gem, which is called unplugging macros. I really recommend that as well. It's like bond macros, but for for anything. Mhmm. But what we do is Wes, extract the abstract syntax tree of the function that you Yarn with use GPU.

We turn it into a very compressed format that we call tiniest, which stands for tiny ESD, like tiny embeddable syntax tree, but we just we figured out that name just for the sake of reserving that name on npm.

So, we inject that metadata into the shader into the user's days bundle. And then at runtime, whenever somebody gives us their function, us, I mean, the the TypeScript API, whenever somebody says, hey. TypeGPU generate a shader for this function, we're we're able to see, okay, did you associate metadata with this function? Okay. Cool. Let me extract that metadata. And then based on that, we generate the shader code. The the details of, like, the how we traverse tree and generate Wizzle from it, it's it's gonna be in my in my talk, hopefully, on YouTube. Yeah. Yeah. Definitely. Someday soon. It's it's definitely, like, a complex process, but, thankfully, like, at the end of the day, the code that you write is pretty similar to the code that ends up on the other end.

And we really wanted to make sure that we don't obstruct away too much of what's happening on the on the underlying platform.

The only thing that really changes is the language that you're writing it in, which to us is really powerful because we can build examples like the jelly slider, built by Konradretsko.

The jelly slider is really good, like, killer Deno. It blew up in a day. We were really surprised, surprised to see it, but it it the reception has been amazing.

And, the only negative comments that we see we saw was, like, now my manager is gonna make me implement this thing into my website. Okay. It has three d all over the web. Yeah. Just to get yeah. And if you want to see all of the errors in your application,

you'll want to check out Sentry at century.io/syntax.

You don't want a production application out there that, well, you have no visibility into in case something is blowing up, and you might not even know it. So head on over to century.io/syntax.

Again, we've been using this tool for a long time, and it totally rules. Alright.

Yeah. I guess speaking of three d, one of the comments that I got on my my video when I talked about TypeScript was wondering about how how does this replace something like three JS, or how does this fit into that ecosystem? Because I think a lot of people's introduction to any kind of graphics programming on the web, for web developers. Might have been through React three fiber or three JS or anything like that. So do you wanna maybe touch on how does this fit into that picture? Yeah. Of course. Of course.

I mean, to to us, web g p a type GPU was always, meant to be a very succinct and and small wrapper on on top of WebGPU.

And therefore, we don't abstract I mean, it's good that we don't abstract away too much, but also the users have to implement a lot from scratch. If you look at the Jelly Slider example, there's a lot of code that goes into it because we have to implement the the rendering logic.

It does ray marching. It does reflections and shadows.

And because we write that ourselves, we Yarn able to control every aspect of it and optimize around it, which is great. But for people that want to have, just plop a three d model on their website, three JS is the way to go. I'm definitely not recommending, like, to you to to build everything from scratch.

But for those that want to delve deeper into, optimizing, and have that low level control over what's actually happening, what code is running on the GPU, I can really recommend starting off from TypeScript and learning, that ecosystem.

But I feel like Type GPU still has a place alongside three JS. We definitely don't wanna, like, replace and build a new three JS from from scratch. Three JS is an amazing ecosystem, and has a lot of community support and resources out there.

So what we're hoping, is that the low level building blocks built by, libraries or by hobbyists that, specialize in graphics programming, can build them using TypeScript and then hook them into Three. Js. We have an upcoming Three. Js integration. Hopefully, by the time this interview is, online, we we would have released the 0.9 version of Type GPU. Yeah. And with it, we have a Three. Js integration where Cool. You can replace any part of your material, with a type GPU function. Cool. So let's say the, like, the lighting logic and everything is still handled by Three. Js, but you just want to change the color of the material, you can just call from, or to TSL, which is the node setup that Three. Js has, pnpm in that type g p function, and you have a TSL node that you can plug into your material.

And, the same goes for compute shaders. If you want a way to write compute shaders in a more, JavaScript way where you have if statements, four loops, and everything, you can do that. You can do from you can do two TSL, pass in the TypeScript function dot compute, and, and you're gonna have a compute shader working and interfacing with three. Js.

So that's that's really exciting. It's it's been in the works for a while, and I'm really happy to see it come together. And hopefully, it's gonna, like, make people more interested or or, have a feeling that they can do more with Three. Js now and delve deeper into those details and and play around with them. Yeah. Yeah. I I I think Type GB will go a long ways to,

reducing just that, like man, every single time I've tried to learn Wes GPU, there is just such a tremendous amount of dread that comes over you when you start to work in it. You're like, oh, this is way I'm way out of my depth Yeah. In this. So, like, yeah, hopefully, that reduces that.

Because, yeah, Three JS, things like that, React three fiber, they're so approachable, but, they're good introductions to wanting to get get deeper with that. One thing that I I really admire about your project is the documentation.

Mhmm. The docs are really great. There's a lot of awesome examples.

It's very readable and parsable, and you can get in and get your hands dirty for me. I'm, like, the type of person that really loves to get in there and tweak values and see what happens.

What was, like, the strategy there in developing docs? Was that always a thought, or is it just something that you personally like in documentation?

I I I mean, I feel like I've I've always liked writing about stuff, but, it's definitely something that's required a kind of a a a ESLint of a mindset.

A little bit of background on on type GPU.

I mean, when when it started, it did have a use case. I was building it for my master thesis, which involved, doing super resolution of, a game running in real time on the local user's machine.

And I want to do it in the browser to just have, a better time working working with with the GPU. And WebGPU was just coming coming up, and I was building up, this API. So Node type GPU, at least in its first form before we were able to assemble a a bigger team, It did have a use case of connecting different libraries together, but once we started working on it, we didn't really have, like, a project or a product that we could use TypeScript in. So there was no, like, driving force to add APIs to Type GPU that would help with, a specific use case. So we were kind of, needing Wes were we kind of needed to create our own product to have a use case for Type GPU. And examples, kind of became that. We were just like, okay. Well, I want to do this using Type GPU. So let's build this. What is missing? Okay. Let's, like, plug in these APIs. Let's create them. Okay. Does this work fine? Okay. Let's create another example.

No. This, like, doesn't generalize that much. Let's let's try to think it through. And I feel like like, we we build a lot of these examples before we ship or stabilize features in type GPU because we want to ensure that, like, it's it's both fun to use and it it works for for those particular use cases.

So that kind of came naturally where, I mean, we, when we shipped Type GPU at first, like, we had a dozen examples or something, right from the start because we wanted to make sure that these APIs worked. And now they're kind of I mean, they're cool to to peruse around, but also when we introduce some breaking changes, they're great, end to end tests. Right? We want to ensure that we don't break anything. So we always run these examples when we change something and then see if anything breaks. And as for the docs, I mean, we really wanted to explain how it works at first, but we kind of noticed that whenever we were describing a new API, when we were having a hard time explaining what it did or it wasn't intuitive to explain, it it was probably a wrong API. Mhmm.

So it was very beneficial to try to explain it to both ourselves, the rest of the team, and to see if it's explainable in, like, a paragraph or or if it's if it, like, if it's easy to learn step by step.

And it led us to a lot of exploration of of other things that maybe are easier to explain. And, yeah, that's that's kind of the driving force. I I feel like docs are usually seen as this, last step of, like, oh, okay. Now I gotta do the docs. Yeah. I I I wrote this, and it works amazing, and now I have to write about it. Yeah.

And I it takes time away from me actually building it. But, I feel like in our case, when we did this mind shift of I I want to build this cool thing, so I have to explain it to my teammates and, like, figure out what APIs I need, then it kind of comes naturally. And, Yeah. We were still pushing forward with that, philosophy, I guess.

So expect more examples. Yeah. More examples to come.

A lot of people talk about GPU, web GPU in general being not just for graphics, but for interfacing with the GPU to do things like AI in browser. Is that a focus of type GPU at all, or is it just something that it enables? Yeah. It it definitely enables that. I mean, we,

we're definitely trying to build out the ecosystem ourselves if we Node it for, say, an example.

Our main approach for, like, building use case specific stuff for TypeScript, is usually Wes create a couple of examples that might have this functionality, and we see what's shared between them.

And that happened for the noise library that we have, which, implements a lot of the basic, like, random functions and and patterns that you can pnpm into your materials or or, shaders.

And with AI, we have one example that does this. We have, a MNIST digit inference. So we can draw a digit, and it's, figures out what that digit is, on your local GPU.

And I feel like once we once we build out more examples that are that are using AI, we're gonna see an abstraction emerge there that we can extract in this into a separate library. Yeah. But we really wanna make sure that type GPU itself doesn't, maybe not get tainted, but, like, doesn't involve itself with the use cases, that it can be used for. Wes really want it to be open for interpretation, I guess. Yeah. Yeah. Yeah. So whenever we do build some use case specific stuff, we release it as a separate library.

And at least to me, it's, it might be useful for people wanting to wanting to create their own libraries that hook into this system Wes, they can look at the simple simpler libraries that build on top of that GPU and use it as a template for for their own, like, code or even their own library if they want to create one.

So it's it both ensures that our API of the core library is flexible enough so that we can build cool libraries on top of it and not have to wait for, like, a feature request for, hey. Can you expose this API for us? Because we need to hook into that by, yeah, by restraining ourselves to only put the essentials and low level stuff in the core library and having to use that limited API set. We expose that same API to the end user, and, I feel like that's that's very powerful for customizability and and extendability.

So, yeah, answering the question directly, we Yarn working on AI stuff, but every AI example is very involved. Usually involves, a lot of research around which model is best to use or even training our own model, and coding the specific layers that need to be a part of it since, at least right now, we have to build all this stuff from scratch.

So it's taking a while. But once we have a couple of these examples, we'll definitely look at,

giving people a better abstraction call. That makes sense. It's it's cool to see how you've architected the library because you import from standard library, and, like, you've kind of separated these things out to their purpose use. So it makes sense that maybe if there's enough of a use case, there might be, like, type GPU AI or something like but it's it's not included in the main package, but you can you can grab it if you need it. Yeah. So we're getting pretty close to the end. Is there anything we haven't asked you that you want to say or anything you'd like to tell the world?

If there's anything I'd like to tell the world, there's so much.

Follow the Syntax podcast. Okay. Appreciate it. Thank you.

Yeah. I I, listen to it every time I, on my way to work.

It's, it's an amazing way to catch up on all the newest JS, ecosystem, news. And, yeah, I'm I'm really excited to get to work, in the open source. I encourage everyone to try out TypeScript and to, to contribute, to look at the docs and and see if it's, something that you'd like to dabble into. I feel like graphics and GPU stuff is having kind of a resurgence on the Wes. Yeah.

And, it's it's signing to be part of that wave, and to, hopefully make it,

make it better and make the ecosystem a bit more approachable. Yeah. Thank you for doing that, making it much more accessible for Yeah. People like me, for sure. Absolutely.

Do you wanna do sick picks? Yeah. Let's do sick picks. Cool. Okay. Yeah. So sick picks for, people who don't know, it's part of the show where we talk about things that we enjoy and like. And, Deno, we've kinda put you on the spot here. I know. Just, ushered you up here. I should've known. So is there anything that you're really just enjoying in life? Could be anything? Pierogies are great. Oh, yeah. Yes. If if if anybody is not into Polish food yet,

try it and you will be.

And go and, come to Krakow. Krakow has a has a has an amazing tech space emerging.

And if you come, let's let's hang out. Let's talk, graphic stuff. Nice. Awesome. Yeah. Cool. So thank you so much for being on. Lastly, big shout out to JS Nation. Thank you for giving us the space and letting us, use this to do this interview. This was great. Yeah. It's been awesome. Thank you so much.