Last week in review #1
ICML reviews, another RL library, and more!
Last Week in Review is a post where I’ll talk about stuff that happened last week in the machine learning community that I’m aware of. Every post won’t be like this, but they’ll happen every once in a while, particularly when I don’t have the bandwidth to publish a large, deep, thoughtful post. If you like it, consider subscribing (it’s free)!
ICML reviews came out
For any unfamiliar readers, ICML (International Conference on Machine Learning) is one of the top AI/ML conferences that happens each year.
A lot of people complained (rightly) on Twitter:
Unfortunately I can’t find a lot of the tweets I read. Most people don’t seem to use the #ICML2021 hashtag. But, there’s a general theme (as there always is) of reviewers complaining about lack of novelty, or that the method is too simple and it’s trivial, or that the result is obvious, or that the authors should have done some experiment the reviewer just came up with but they didn’t, or… You get the idea.
My group got kind of lucky, we only got one reviewer complaining about novelty! They complained that our work is like blah et al when we are blah et al and we’re intentionally extending our earlier work. But that’s nothing new.
Good luck to everyone dealing with review rebuttals right now. The fact that reviews and rebuttals seem to go poorly every conference really makes me wonder if the system is broken. Maybe it would be better to get rid of the rebuttal period altogether?
Facebook AI releases rlstructures library
Last week, Facebook open-sourced a new RL framework called “rlstructures”. It’s on Github here and the documentation is here. I’m curious about this package and want to try it out. It’s a very young package and in early versioning (current version appears to be v0.2) so I’m sure the API will change some.
It seems like the maintainers are currently active, they mention recently changing the API in the documentation, and it says in their README that they made the API changes in response to user feedback - so that’s great!
What I’m excited about
This framework focuses on making it easier to parallelize your agents and environments, while trying to stay out of the way of the implementation of the agent and loss function. This is great because it can be really frustrating and challenging to write your own multithreaded code, so using something like this has a good chance of making your life easier. It also should be pretty flexible, since the framework is hands-off on the agent implementation side of things.
What I don’t love
They describe it as “a way to i) simulate multiple policies, multiple models and multiple environments simultaneously at scale ii) define complex loss functions and iii) quickly implement various policy architectures.” but everything I see in the documentation is only targeted towards parallelization, which is point one in the sentence. So I’m not sure how they’re facilitating the other things they mention. The API also looks a little rough to me. Although this is to be expected in such a young package. I also may try it out and end up deciding that the API is great. I am bummed that it’s PyTorch only though. While I am a PyTorch user, it’s nice to have flexibility to use JAX or TF if you really want to.
The Minecraft Open-Endedness Challenge
Open-endedness is a fascinating problem, described by Kenneth Stanley and others as “a process that tirelessly invents ever-greater complexity and novelty across incomprehensible spans of time.” Evolution is an open-ended process, and Stanley points out that if you think computationally about evolution, you realize that it has generated all of the life on Earth in one run of the algorithm. This “one run” has played out over billions of years to give us the amazing diversity of life that we see today. Evolution also invented human intelligence, so open-ended algorithms may be a path to human-level AI. Read Stanley’s article here.
The Minecraft Open-Endedness Challenge aims to inspire people to create open-ended algorithms and will be scored on Divergence, Diversity, Complexity, Ecological Interactions, and Life-like Properties. The challenge is sponsored by OpenAI and modl.ai and is occurring as part of GECCO 2021. Here’s some inspirational videos of relevant work done using the EvoCraft API that the challenge uses.
If you’re curious for more, check out the challenge page.
The Robot Brains podcast by Pieter Abbeel
Pieter Abbeel is a top researcher in AI for robotics and just last week he released his first podcast episode. I’m a reinforcement learning and robotics lover, so I had to go listen to it. It’s an illuminating episode, and I think Pieter asks good questions. His guest is Andrej Karpathy, another rockstar in the AI/ML world, and Andrej and Pieter have a very interesting and fruitful discussion about Andrej’s past experiences and his life as the Director of AI at Tesla. I highly recommend it if you’re curious about these things. I’m excited to see what episodes Pieter has coming out next!
Thanks for reading! If you enjoyed it, consider subscribing (it’s free)!
Please share your thoughts with me in the comments below.