* "Radosgw installation and administration" docs [not found] <4FD71854.6060503@hastexo.com> @ 2012-06-12 10:44 ` Florian Haas 2012-06-12 16:47 ` Yehuda Sadeh 0 siblings, 1 reply; 9+ messages in thread From: Florian Haas @ 2012-06-12 10:44 UTC (permalink / raw) To: ceph-devel Hi everyone, I have a long flight ahead of me later this week and plan to be spending some time on http://ceph.com/docs/master/ops/radosgw/ -- which currently happens to be a bit, ahem, sparse. There's currently not a lot of documentation on radosgw, and some of it is inconsistent, so if one of the devs could answer the following questions, I can put them in a more comprehensive document that should make radosgw easier to set up and run. 1. Apache rewrite rule Is the Apache configuration example listed in the man page correct and authoritative? Specifically, it seems unclear to me whether the rewrite engine rule: (RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*) /s3gw.fcgi?page=$1¶ms=$2&%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]) ... is expected to work only for compatibility with S3 clients, or whether this rewrite rule is also for Swift clients. 2. FastCGI wrapper The radosgw man page says it should be "exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway", whereas the Wiki (http://ceph.com/wiki/RADOS_Gateway) omits the -n option. I didn't get it to work without the -n option, so is it safe to say that it is required? 3. Apache/radosgw daemon/FastCGI wrapper interaction Is it safe to say that we always need all three of these? The man page indicates so, the Wiki makes no mention of the daemon started by the init script. 4. FastCGI configuration directives The man page mentions: FastCgiExternalServer /var/www/s3gw.fcgi -socket /tmp/radosgw.sock The Wiki says: FastCgiWrapper /var/www/s3gw.fcgi FastCgiServer /usr/bin/radosgw https://github.com/ceph/teuthology/blob/master/teuthology/task/apache.conf (which was mentioned as an additional reference on IRC at some point) says: FastCgiIPCDir /tmp/cephtest/apache/tmp/fastcgi_sock FastCgiExternalServer /tmp/cephtest/apache/htdocs/rgw.fcgi -socket rgw_sock Which of these is required/preferred? -socket option or not? Wrapper, Server or ExternalServer? IPCDir? 5. Logging What's the preferred way of adding debug logging for radosgw? https://github.com/ceph/teuthology/blob/master/teuthology/task/apache.conf mentions: SetEnv RGW_LOG_LEVEL 20 SetEnv RGW_PRINT_CONTINUE yes SetEnv RGW_SHOULD_LOG yes ... but it's unclear to me whether this is still current (I found no trace of those envars in the source, but maybe I was looking in the wrong place). https://github.com/ceph/ceph/commit/452b1248a68f743ad55641722da80e3fd5ad2ae9 touched the "debug rgw" option. If that is the preferred way of doing things now, where should you set this? In ceph.conf, in the [client.radosgw.<name>] section? Also, for each of these, where would the logging output end up? /var/log/ceph? Apache error log? If so, only if the Apache LogLevel is more verbose than info? Syslog? 6. Swift API: Keys Is it correct to assume that for any Swift client to work, we must set a Swift key for the user, like so? radosgw-admin key create --key-type=swift --uid=<user> If so, is the secret_key that that creates for the user: "swift_keys": [ { "user": "<user>", "secret_key": "<longbase64hash>"}]} ... the same key that the swift command line client expects to be set with th -K option? 7. Swift API: swift user name When we call "swift -U <user>", is that the verbatim user_id that we've defined with "radosgw-admin user create --uid=<user_id>"? Or do we need to set a prefix? Or define a separate Swift user ID? 8. Swift API: authentication version When radosgw acts as the auth server for a Swift request, is it correct to say that only v1.0 Swift authentication is supported, not v2.0? 9. Swift API: authentication URL What's the correct Swift authentication URL for "swift -A <url>"? It seems like it's "http://<rgw hostname>:<port>/auth", but confirmation would help. 10. radosgw "OpenStack user" information From the radosgw-admin man page: --os-user=group:name The OpenStack user (only needed for use with OpenStack) --os-secret=key The OpenStack key What's this meant to be used for? Keystone authentication? If so, is there anything else that needs to be done for Keystone to work with this, such as add an endpoint URI? Please feel free to point me to existing documentation where it exists. Your help is much appreciated. Thanks! Cheers, Florian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: "Radosgw installation and administration" docs 2012-06-12 10:44 ` "Radosgw installation and administration" docs Florian Haas @ 2012-06-12 16:47 ` Yehuda Sadeh 2012-06-12 18:11 ` Florian Haas 0 siblings, 1 reply; 9+ messages in thread From: Yehuda Sadeh @ 2012-06-12 16:47 UTC (permalink / raw) To: Florian Haas; +Cc: ceph-devel On Tue, Jun 12, 2012 at 3:44 AM, Florian Haas <florian@hastexo.com> wrote: > Hi everyone, > > I have a long flight ahead of me later this week and plan to be > spending some time on http://ceph.com/docs/master/ops/radosgw/ -- which > currently happens to be a bit, ahem, sparse. > > There's currently not a lot of documentation on radosgw, and some of it > is inconsistent, so if one of the devs could answer the following > questions, I can put them in a more comprehensive document that should > make radosgw easier to set up and run. > > 1. Apache rewrite rule > > Is the Apache configuration example listed in the man page correct and > authoritative? Specifically, it seems unclear to me whether the > rewrite engine rule: > > (RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*) > /s3gw.fcgi?page=$1¶ms=$2&%{QUERY_STRING} We currently use a slightly different rule: RewriteRule ^/(.*) /radosgw.fcgi?params=$1&%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] > [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]) > > ... is expected to work only for compatibility with S3 clients, or > whether this rewrite rule is also for Swift clients. Not really needed for Swift. It's required for passing in the HTTP_AUTHORIZATION env, however, Swift uses a different field which is not filtered out by apache. > > > 2. FastCGI wrapper > > The radosgw man page says it should be "exec /usr/bin/radosgw -c > /etc/ceph/ceph.conf -n client.radosgw.gateway", whereas the Wiki > (http://ceph.com/wiki/RADOS_Gateway) omits the -n option. I didn't get > it to work without the -n option, so is it safe to say that it is required? -n is required for specifying the ceph user that the gateway would use. Without it it'd use client.admin is the default. > > > 3. Apache/radosgw daemon/FastCGI wrapper interaction > > Is it safe to say that we always need all three of these? The man page indicates > so, the Wiki makes no mention of the daemon started by the init script. The wrapper is not needed if not using apache for spawning the radosgw processes. E.g., when using the FastCgiExternalServer param: FastCgiExternalServer /var/www/radosgw.fcgi -socket /var/run/ceph/radosgw.client.radosgw > > > 4. FastCGI configuration directives > > The man page mentions: > FastCgiExternalServer /var/www/s3gw.fcgi -socket /tmp/radosgw.sock > > The Wiki says: > FastCgiWrapper /var/www/s3gw.fcgi > FastCgiServer /usr/bin/radosgw > > https://github.com/ceph/teuthology/blob/master/teuthology/task/apache.conf > (which was mentioned as an additional reference on IRC at some point) says: > FastCgiIPCDir /tmp/cephtest/apache/tmp/fastcgi_sock > FastCgiExternalServer /tmp/cephtest/apache/htdocs/rgw.fcgi -socket rgw_sock > > Which of these is required/preferred? -socket option or not? Wrapper, > Server or ExternalServer? IPCDir? > Either one is required. We prefer using the external server option. We found out that letting apache (or the fastcgi process manager) managing was sub-optimal and was introducing high latencies. > > 5. Logging > > What's the preferred way of adding debug logging for radosgw? > > https://github.com/ceph/teuthology/blob/master/teuthology/task/apache.conf > mentions: > > SetEnv RGW_LOG_LEVEL 20 > SetEnv RGW_PRINT_CONTINUE yes > SetEnv RGW_SHOULD_LOG yes All are obsolete and defunct, and have a corresponding ceph.conf conf: debug rgw = 20 rgw print continue = true rgw should log = true the latter will be replaced soon by: rgw enable usage log = true Note that only the 'debug rgw' option is really related to debug logs. The 'rgw print continue' option is a badly named option to control the use of 100-continue (should the radosgw 'print' -- as in FCGX_FPrintF -- the 100-continue when it should?). This can only work with a modified mod_fastcgi that supports that. The 'rgw should log' option sets whether we log each user operation to the dedicated pool (so that it can be analyzed later on for billing, etc.) > > ... but it's unclear to me whether this is still current (I found no > trace of those envars in the source, but maybe I was looking in the > wrong place). > > https://github.com/ceph/ceph/commit/452b1248a68f743ad55641722da80e3fd5ad2ae9 > touched the "debug rgw" option. If that is the preferred way of doing > things now, where should you set this? In ceph.conf, in the > [client.radosgw.<name>] section? Either under the global section, or [client], or [client.radosgw.<name>]. Depends on how you organize your conf. > > Also, for each of these, where would the logging output end up? > /var/log/ceph? Apache error log? If so, only if the Apache LogLevel is > more verbose than info? Syslog? The debug log would end up wherever you specified in the 'log file' config option. > > > 6. Swift API: Keys > > Is it correct to assume that for any Swift client to work, we must set a > Swift key for the user, like so? > > radosgw-admin key create --key-type=swift --uid=<user> > > If so, is the secret_key that that creates for the user: > > "swift_keys": [ > { "user": "<user>", > "secret_key": "<longbase64hash>"}]} > > > ... the same key that the swift command line client expects to be set > with th -K option? Yes. > > > 7. Swift API: swift user name > > When we call "swift -U <user>", is that the verbatim user_id that we've > defined with "radosgw-admin user create --uid=<user_id>"? Or do we need > to set a prefix? Or define a separate Swift user ID? > In swift the terminology is a bit different. There is the account and under that there is the user. Since we already have a 'user' (which is actually the swift account), we created a 'subuser'. So a one liner user and swift-subuser creation would be as follows: # radosgw-admin user create --subuser=yehuda:yehuda1 --display-name=Yehuda --key-type=swift --access=full { "user_id": "yehuda", "rados_uid": 0, "display_name": "Yehuda", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [ { "id": "yehuda:yehuda1", "permissions": "full-control"}], "keys": [], "swift_keys": [ { "user": "yehuda:yehuda1", "secret_key": "7TD5f2QrwxkCnhthwowC4d9uEJ4mnX8nGsXjmnW8"}]} The --access=full will give the subuser a full access to the account; other options would be read, write, readwrite. > > 8. Swift API: authentication version > > When radosgw acts as the auth server for a Swift request, is it correct > to say that only v1.0 Swift authentication is supported, not v2.0? Yeah. Currently radosgw serves as v1.0 authenticator. > > > 9. Swift API: authentication URL > > What's the correct Swift authentication URL for "swift -A <url>"? It > seems like it's "http://<rgw hostname>:<port>/auth", but confirmation > would help. Confirmed. > > 10. radosgw "OpenStack user" information > > From the radosgw-admin man page: > --os-user=group:name > The OpenStack user (only needed for use with OpenStack) > --os-secret=key > The OpenStack key Obsolete. That was the old way to configure swift users. > > What's this meant to be used for? Keystone authentication? If so, is > there anything else that needs to be done for Keystone to work with > this, such as add an endpoint URI? iirc, the swift protocol provides the endpoint URI in the HTTP header, so if the token was generated by another swift authenticator, we'd try to authenticate against it. I'm not familiar with keystone, and whether it's supposed to work with it. > > Please feel free to point me to existing documentation where it > exists. Your help is much appreciated. Thanks! > That's my radosgw ceph.conf that I'm using in my test environment. [client] admin socket = /tmp/radosgw.adsock debug ms = 1 rgw socket path = /tmp/.radosgw.sock auth supported = none log file = /var/log/radosgw/radosgw.log debug rgw = 20 rgw cache enabled = 1 ; rgw swift url = http://skinny ; rgw swift url prefix = swift rgw dns name = skinny rgw cache lru size = 1000 rgw enable ops log = false ; rgw print continue = false [mon.a] host = swab mon addr = 192.168.106.223:14090 That's my apache site conf: FastCgiExternalServer /var/www/web1/web/radosgw.fcgi -socket /tmp/.radosgw.sock <VirtualHost *:80> ServerName skinny.ops.newdream.net ServerAlias skinny ServerAdmin webmaster@example1.com DocumentRoot /var/www/web1/web/ #turn engine on RewriteEngine On #following is important for S3/rados RewriteRule ^/(.*) /radosgw.fcgi?params=$1&%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] <IfModule mod_fastcgi.c> SuexecUserGroup web1 web1 # PHP_Fix_Pathinfo_Enable 1 <Directory /var/www/web1/web/> Options +ExecCGI AllowOverride None SetHandler fastcgi-script Order allow,deny Allow from all AuthBasicAuthoritative Off </Directory> </IfModule> AllowEncodedSlashes On ErrorLog /var/log/apache2/error.log CustomLog /var/log/apache2/access.log combined ServerSignature Off # DumpIOInput On # DumpIOOutput On </VirtualHost> Thanks, Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: "Radosgw installation and administration" docs 2012-06-12 16:47 ` Yehuda Sadeh @ 2012-06-12 18:11 ` Florian Haas 2012-06-12 18:54 ` Yehuda Sadeh 0 siblings, 1 reply; 9+ messages in thread From: Florian Haas @ 2012-06-12 18:11 UTC (permalink / raw) To: Yehuda Sadeh; +Cc: ceph-devel Hi Yehuda, thanks, that resolved a lot of questions for me. A few follow-up comments below: On 06/12/12 18:47, Yehuda Sadeh wrote: > On Tue, Jun 12, 2012 at 3:44 AM, Florian Haas <florian@hastexo.com> wrote: >> Hi everyone, >> >> I have a long flight ahead of me later this week and plan to be >> spending some time on http://ceph.com/docs/master/ops/radosgw/ -- which >> currently happens to be a bit, ahem, sparse. >> >> There's currently not a lot of documentation on radosgw, and some of it >> is inconsistent, so if one of the devs could answer the following >> questions, I can put them in a more comprehensive document that should >> make radosgw easier to set up and run. >> >> 1. Apache rewrite rule >> >> Is the Apache configuration example listed in the man page correct and >> authoritative? Specifically, it seems unclear to me whether the >> rewrite engine rule: >> >> (RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*) >> /s3gw.fcgi?page=$1¶ms=$2&%{QUERY_STRING} > > We currently use a slightly different rule: > > RewriteRule ^/(.*) > /radosgw.fcgi?params=$1&%{QUERY_STRING} > [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] Could you explain what happened to "page"? >> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]) >> >> ... is expected to work only for compatibility with S3 clients, or >> whether this rewrite rule is also for Swift clients. > > Not really needed for Swift. It's required for passing in the > HTTP_AUTHORIZATION env, however, Swift uses a different field which is > not filtered out by apache. OK. >> 2. FastCGI wrapper >> >> The radosgw man page says it should be "exec /usr/bin/radosgw -c >> /etc/ceph/ceph.conf -n client.radosgw.gateway", whereas the Wiki >> (http://ceph.com/wiki/RADOS_Gateway) omits the -n option. I didn't get >> it to work without the -n option, so is it safe to say that it is required? > > -n is required for specifying the ceph user that the gateway would > use. Without it it'd use client.admin is the default. OK. >> 3. Apache/radosgw daemon/FastCGI wrapper interaction >> >> Is it safe to say that we always need all three of these? The man page indicates >> so, the Wiki makes no mention of the daemon started by the init script. > > The wrapper is not needed if not using apache for spawning the radosgw > processes. E.g., when using the FastCgiExternalServer param: > > FastCgiExternalServer /var/www/radosgw.fcgi -socket > /var/run/ceph/radosgw.client.radosgw > >> 4. FastCGI configuration directives >> >> The man page mentions: >> FastCgiExternalServer /var/www/s3gw.fcgi -socket /tmp/radosgw.sock >> >> The Wiki says: >> FastCgiWrapper /var/www/s3gw.fcgi >> FastCgiServer /usr/bin/radosgw >> >> https://github.com/ceph/teuthology/blob/master/teuthology/task/apache.conf >> (which was mentioned as an additional reference on IRC at some point) says: >> FastCgiIPCDir /tmp/cephtest/apache/tmp/fastcgi_sock >> FastCgiExternalServer /tmp/cephtest/apache/htdocs/rgw.fcgi -socket rgw_sock >> >> Which of these is required/preferred? -socket option or not? Wrapper, >> Server or ExternalServer? IPCDir? >> > > Either one is required. We prefer using the external server option. We > found out that letting apache (or the fastcgi process manager) > managing was sub-optimal and was introducing high latencies. OK, I'm sticking to FastCgiExternalServer then. >> 5. Logging >> >> What's the preferred way of adding debug logging for radosgw? >> >> https://github.com/ceph/teuthology/blob/master/teuthology/task/apache.conf >> mentions: >> >> SetEnv RGW_LOG_LEVEL 20 >> SetEnv RGW_PRINT_CONTINUE yes >> SetEnv RGW_SHOULD_LOG yes > > All are obsolete and defunct, and have a corresponding ceph.conf conf: > > debug rgw = 20 > rgw print continue = true > rgw should log = true > > the latter will be replaced soon by: > > rgw enable usage log = true > > Note that only the 'debug rgw' option is really related to debug logs. > The 'rgw print continue' option is a badly named option to control the > use of 100-continue (should the radosgw 'print' -- as in FCGX_FPrintF > -- the 100-continue when it should?). This can only work with a > modified mod_fastcgi that supports that. > The 'rgw should log' option sets whether we log each user operation to > the dedicated pool (so that it can be analyzed later on for billing, > etc.) Yep. I was really only looking for what "debug rgw" does, and got confused by the FastCGI envars. >> ... but it's unclear to me whether this is still current (I found no >> trace of those envars in the source, but maybe I was looking in the >> wrong place). >> >> https://github.com/ceph/ceph/commit/452b1248a68f743ad55641722da80e3fd5ad2ae9 >> touched the "debug rgw" option. If that is the preferred way of doing >> things now, where should you set this? In ceph.conf, in the >> [client.radosgw.<name>] section? > > Either under the global section, or [client], or > [client.radosgw.<name>]. Depends on how you organize your conf. OK. >> Also, for each of these, where would the logging output end up? >> /var/log/ceph? Apache error log? If so, only if the Apache LogLevel is >> more verbose than info? Syslog? > > > The debug log would end up wherever you specified in the 'log file' > config option. ... or syslog, if log file = "" and syslog = true. (iirc) >> 6. Swift API: Keys >> >> Is it correct to assume that for any Swift client to work, we must set a >> Swift key for the user, like so? >> >> radosgw-admin key create --key-type=swift --uid=<user> >> >> If so, is the secret_key that that creates for the user: >> >> "swift_keys": [ >> { "user": "<user>", >> "secret_key": "<longbase64hash>"}]} >> >> >> ... the same key that the swift command line client expects to be set >> with th -K option? > > Yes. OK, but I realized that you apparently have to create a separate key when creating a sub-user. Is that correct? Or is there a way for sub-users to "inherit" the keys defined for their parent user? >> 7. Swift API: swift user name >> >> When we call "swift -U <user>", is that the verbatim user_id that we've >> defined with "radosgw-admin user create --uid=<user_id>"? Or do we need >> to set a prefix? Or define a separate Swift user ID? >> > > In swift the terminology is a bit different. There is the account and > under that there is the user. Since we already have a 'user' (which is > actually the swift account), we created a 'subuser'. So a one liner > user and swift-subuser creation would be as follows: > > # radosgw-admin user create --subuser=yehuda:yehuda1 > --display-name=Yehuda --key-type=swift --access=full It seems there is some magic involved so that if you do "user create", set --subuser=<prefix>:<sub> and don't set --uid, it creates a new parent user named <prefix>. Is this meant to be stable? Or is the supported way of doing things to always first create a user, and then use "subuser create" to create the subuser? > { "user_id": "yehuda", > "rados_uid": 0, > "display_name": "Yehuda", > "email": "", > "suspended": 0, > "max_buckets": 1000, > "subusers": [ > { "id": "yehuda:yehuda1", > "permissions": "full-control"}], > "keys": [], > "swift_keys": [ > { "user": "yehuda:yehuda1", > "secret_key": "7TD5f2QrwxkCnhthwowC4d9uEJ4mnX8nGsXjmnW8"}]} > > The --access=full will give the subuser a full access to the account; > other options would be read, write, readwrite. OK. >> 8. Swift API: authentication version >> >> When radosgw acts as the auth server for a Swift request, is it correct >> to say that only v1.0 Swift authentication is supported, not v2.0? > > Yeah. Currently radosgw serves as v1.0 authenticator. So I figured. >> 9. Swift API: authentication URL >> >> What's the correct Swift authentication URL for "swift -A <url>"? It >> seems like it's "http://<rgw hostname>:<port>/auth", but confirmation >> would help. > > Confirmed. Thanks. >> 10. radosgw "OpenStack user" information >> >> From the radosgw-admin man page: >> --os-user=group:name >> The OpenStack user (only needed for use with OpenStack) >> --os-secret=key >> The OpenStack key > > Obsolete. That was the old way to configure swift users. OK. Should this be removed from the man page then? >> What's this meant to be used for? Keystone authentication? If so, is >> there anything else that needs to be done for Keystone to work with >> this, such as add an endpoint URI? > > iirc, the swift protocol provides the endpoint URI in the HTTP header, > so if the token was generated by another swift authenticator, we'd try > to authenticate against it. I'm not familiar with keystone, and > whether it's supposed to work with it. OK. >> Please feel free to point me to existing documentation where it >> exists. Your help is much appreciated. Thanks! >> > > That's my radosgw ceph.conf that I'm using in my test environment. > > [client] > admin socket = /tmp/radosgw.adsock > debug ms = 1 > rgw socket path = /tmp/.radosgw.sock > auth supported = none Silly question: If "auth supported = none", is it still required to run the ceph-authtool and ceph-auth commands specified in radosgw(8)? > log file = /var/log/radosgw/radosgw.log > debug rgw = 20 > rgw cache enabled = 1 > ; rgw swift url = http://skinny > ; rgw swift url prefix = swift I ran across this option sifting through src/rgw, can you explain what the URL prefix is used for? > rgw dns name = skinny > rgw cache lru size = 1000 > rgw enable ops log = false > ; rgw print continue = false > > [mon.a] > host = swab > mon addr = 192.168.106.223:14090 > > That's my apache site conf: > > FastCgiExternalServer /var/www/web1/web/radosgw.fcgi -socket /tmp/.radosgw.sock > > <VirtualHost *:80> > ServerName skinny.ops.newdream.net > ServerAlias skinny > ServerAdmin webmaster@example1.com > DocumentRoot /var/www/web1/web/ > > #turn engine on > RewriteEngine On > > #following is important for S3/rados > RewriteRule ^/(.*) > /radosgw.fcgi?params=$1&%{QUERY_STRING} > [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] > > <IfModule mod_fastcgi.c> > SuexecUserGroup web1 web1 > # PHP_Fix_Pathinfo_Enable 1 > <Directory /var/www/web1/web/> > Options +ExecCGI > AllowOverride None > SetHandler fastcgi-script > Order allow,deny > Allow from all > AuthBasicAuthoritative Off > </Directory> > </IfModule> > > AllowEncodedSlashes On > > ErrorLog /var/log/apache2/error.log > CustomLog /var/log/apache2/access.log combined > ServerSignature Off > > # DumpIOInput On > # DumpIOOutput On > > </VirtualHost> Thanks for your insight so far. I take that as a "WTFM". :) Cheers, Florian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: "Radosgw installation and administration" docs 2012-06-12 18:11 ` Florian Haas @ 2012-06-12 18:54 ` Yehuda Sadeh 2012-06-28 7:50 ` Florian Haas 2012-07-01 20:22 ` Chuanyu 0 siblings, 2 replies; 9+ messages in thread From: Yehuda Sadeh @ 2012-06-12 18:54 UTC (permalink / raw) To: Florian Haas; +Cc: ceph-devel On Tue, Jun 12, 2012 at 11:11 AM, Florian Haas <florian@hastexo.com> wrote: > Hi Yehuda, > > thanks, that resolved a lot of questions for me. A few follow-up > comments below: > >> >> We currently use a slightly different rule: >> >> RewriteRule ^/(.*) >> /radosgw.fcgi?params=$1&%{QUERY_STRING} >> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] > > Could you explain what happened to "page"? Not really. I don't remember, was probably necessary originally and now it's not. Looking at the code, I think you can also drop the params=$1 part: RewriteRule ^/(.*) /radosgw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] > >>> Also, for each of these, where would the logging output end up? >>> /var/log/ceph? Apache error log? If so, only if the Apache LogLevel is >>> more verbose than info? Syslog? >> >> >> The debug log would end up wherever you specified in the 'log file' >> config option. > > ... or syslog, if log file = "" and syslog = true. (iirc) Yeah. Whichever ceph logging scheme you're using. > >>> 6. Swift API: Keys >>> >>> Is it correct to assume that for any Swift client to work, we must set a >>> Swift key for the user, like so? >>> >>> radosgw-admin key create --key-type=swift --uid=<user> >>> >>> If so, is the secret_key that that creates for the user: >>> >>> "swift_keys": [ >>> { "user": "<user>", >>> "secret_key": "<longbase64hash>"}]} >>> >>> >>> ... the same key that the swift command line client expects to be set >>> with th -K option? >> >> Yes. > > OK, but I realized that you apparently have to create a separate key > when creating a sub-user. Is that correct? Or is there a way for > sub-users to "inherit" the keys defined for their parent user? > >>> 7. Swift API: swift user name >>> >>> When we call "swift -U <user>", is that the verbatim user_id that we've >>> defined with "radosgw-admin user create --uid=<user_id>"? Or do we need >>> to set a prefix? Or define a separate Swift user ID? >>> >> >> In swift the terminology is a bit different. There is the account and >> under that there is the user. Since we already have a 'user' (which is >> actually the swift account), we created a 'subuser'. So a one liner >> user and swift-subuser creation would be as follows: >> >> # radosgw-admin user create --subuser=yehuda:yehuda1 >> --display-name=Yehuda --key-type=swift --access=full > > It seems there is some magic involved so that if you do "user create", > set --subuser=<prefix>:<sub> and don't set --uid, it creates a new > parent user named <prefix>. Is this meant to be stable? Or is the The <user>:<subuser> notation is stable. > supported way of doing things to always first create a user, and then > use "subuser create" to create the subuser? Both are supported, but note that the 'user create' command requires a display-name to be specified, whereas the 'subuser create' doesn't. We can change that later and only require the display-name if the user does not exist, but at the moment that's how it works. > >> { "user_id": "yehuda", >> "rados_uid": 0, >> "display_name": "Yehuda", >> "email": "", >> "suspended": 0, >> "max_buckets": 1000, >> "subusers": [ >> { "id": "yehuda:yehuda1", >> "permissions": "full-control"}], >> "keys": [], >> "swift_keys": [ >> { "user": "yehuda:yehuda1", >> "secret_key": "7TD5f2QrwxkCnhthwowC4d9uEJ4mnX8nGsXjmnW8"}]} >> > >>> 10. radosgw "OpenStack user" information >>> >>> From the radosgw-admin man page: >>> --os-user=group:name >>> The OpenStack user (only needed for use with OpenStack) >>> --os-secret=key >>> The OpenStack key >> >> Obsolete. That was the old way to configure swift users. > > OK. Should this be removed from the man page then? Yeah, should be updated. > > Silly question: If "auth supported = none", is it still required to run > the ceph-authtool and ceph-auth commands specified in radosgw(8)? Not for setting up radosgw. >> log file = /var/log/radosgw/radosgw.log >> debug rgw = 20 >> rgw cache enabled = 1 >> ; rgw swift url = http://skinny >> ; rgw swift url prefix = swift > > I ran across this option sifting through src/rgw, can you explain what > the URL prefix is used for? When authenticating the client, the swift_url and swift_prefix are concatenated and passed to the client as the storage URL, along with the token. Thanks, Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: "Radosgw installation and administration" docs 2012-06-12 18:54 ` Yehuda Sadeh @ 2012-06-28 7:50 ` Florian Haas 2012-07-01 20:22 ` Chuanyu 1 sibling, 0 replies; 9+ messages in thread From: Florian Haas @ 2012-06-28 7:50 UTC (permalink / raw) To: Yehuda Sadeh; +Cc: ceph-devel Hi Yehuda, Not sure if you've seen this; there's a pull request waiting that populates the radosgw admin guide. https://github.com/ceph/ceph/pull/15 Cheers, Florian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: "Radosgw installation and administration" docs 2012-06-12 18:54 ` Yehuda Sadeh 2012-06-28 7:50 ` Florian Haas @ 2012-07-01 20:22 ` Chuanyu 2012-07-02 7:06 ` Florian Haas ` (2 more replies) 1 sibling, 3 replies; 9+ messages in thread From: Chuanyu @ 2012-07-01 20:22 UTC (permalink / raw) To: ceph-devel Yehuda Sadeh <yehuda <at> inktank.com> writes: > > On Tue, Jun 12, 2012 at 11:11 AM, Florian Haas <florian <at> hastexo.com> wrote: > > Hi Yehuda, > > > > thanks, that resolved a lot of questions for me. A few follow-up > > comments below: > > > >> > >> We currently use a slightly different rule: > >> > >> RewriteRule ^/(.*) > >> /radosgw.fcgi?params=$1&%{QUERY_STRING} > >> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] > > > > Could you explain what happened to "page"? > > Not really. I don't remember, was probably necessary originally and > now it's not. Looking at the code, I think you can also drop the > params=$1 part: > > RewriteRule ^/(.*) /radosgw.fcgi?%{QUERY_STRING} > [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] > > > > >>> Also, for each of these, where would the logging output end up? > >>> /var/log/ceph? Apache error log? If so, only if the Apache LogLevel is > >>> more verbose than info? Syslog? > >> > >> > >> The debug log would end up wherever you specified in the 'log file' > >> config option. > > > > ... or syslog, if log file = "" and syslog = true. (iirc) > > Yeah. Whichever ceph logging scheme you're using. > > > > >>> 6. Swift API: Keys > >>> > >>> Is it correct to assume that for any Swift client to work, we must set a > >>> Swift key for the user, like so? > >>> > >>> radosgw-admin key create --key-type=swift --uid=<user> > >>> > >>> If so, is the secret_key that that creates for the user: > >>> > >>> "swift_keys": [ > >>> { "user": "<user>", > >>> "secret_key": "<longbase64hash>"}]} > >>> > >>> > >>> ... the same key that the swift command line client expects to be set > >>> with th -K option? > >> > >> Yes. > > > > OK, but I realized that you apparently have to create a separate key > > when creating a sub-user. Is that correct? Or is there a way for > > sub-users to "inherit" the keys defined for their parent user? > > > >>> 7. Swift API: swift user name > >>> > >>> When we call "swift -U <user>", is that the verbatim user_id that we've > >>> defined with "radosgw-admin user create --uid=<user_id>"? Or do we need > >>> to set a prefix? Or define a separate Swift user ID? > >>> > >> > >> In swift the terminology is a bit different. There is the account and > >> under that there is the user. Since we already have a 'user' (which is > >> actually the swift account), we created a 'subuser'. So a one liner > >> user and swift-subuser creation would be as follows: > >> > >> # radosgw-admin user create --subuser=yehuda:yehuda1 > >> --display-name=Yehuda --key-type=swift --access=full > > > > It seems there is some magic involved so that if you do "user create", > > set --subuser=<prefix>:<sub> and don't set --uid, it creates a new > > parent user named <prefix>. Is this meant to be stable? Or is the > > The <user>:<subuser> notation is stable. > > > supported way of doing things to always first create a user, and then > > use "subuser create" to create the subuser? > > Both are supported, but note that the 'user create' command requires a > display-name to be specified, whereas the 'subuser create' doesn't. We > can change that later and only require the display-name if the user > does not exist, but at the moment that's how it works. > > > > >> { "user_id": "yehuda", > >> "rados_uid": 0, > >> "display_name": "Yehuda", > >> "email": "", > >> "suspended": 0, > >> "max_buckets": 1000, > >> "subusers": [ > >> { "id": "yehuda:yehuda1", > >> "permissions": "full-control"}], > >> "keys": [], > >> "swift_keys": [ > >> { "user": "yehuda:yehuda1", > >> "secret_key": "7TD5f2QrwxkCnhthwowC4d9uEJ4mnX8nGsXjmnW8"}]} > >> > > Hi Yehuda, Florian, I follow the wiki, and steps which you discussed, construct my ceph system with rados gateway, and I can use libs3 to upload file via radosgw, (thanks a lot!) but got "405 Method Not Allowed" when I use swift, $ swift -v -A http://s3.paca.tw:80/auth -U paca:paca1 -K UoJO4nFgdAoX+9nEftElIY+AMmDIkcrUBkycNKPA stat Auth GET failed: http://s3.paca.tw:80/auth/tokens 405 Method Not Allowed ( Because there has no test step on wiki, I follow the Florian's question, and guess the test command is above ?!) my radosgw-admin config: $ radosgw-admin user info --uid=paca { "user_id": "paca", "rados_uid": 0, "display_name": "chuanyu", "email": "chuanyu@cs.nctu.edu.tw", "suspended": 0, "subusers": [ { "id": "paca:paca1", "permissions": "<none>"}], "keys": [ { "user": "paca", "access_key": "DS932H4EI9HK7I1CTDNF", "secret_key": "Rn\/5FqHzRPZFN6f9R\/LuTqvG0AYjbHtrurrGydVk"}], "swift_keys": [ { "user": "paca:paca1", "secret_key": "UoJO4nFgdAoX+9nEftElIY+AMmDIkcrUBkycNKPA"}]} ceph.conf: [client.radosgw.gateway] host = volume keyring = /etc/ceph/keyring/radosgw.gateway.keyring rgw socket path = /var/run/ceph/rgw.sock log file = "" syslog = true debug rgw = 20 my log: http://pastebin.com/rhGhATmv Any advice would be appreciate! Tthanks, Chuanyu > >>> 10. radosgw "OpenStack user" information > >>> > >>> From the radosgw-admin man page: > >>> --os-user=group:name > >>> The OpenStack user (only needed for use with OpenStack) > >>> --os-secret=key > >>> The OpenStack key > >> > >> Obsolete. That was the old way to configure swift users. > > > > OK. Should this be removed from the man page then? > > Yeah, should be updated. > > > > > Silly question: If "auth supported = none", is it still required to run > > the ceph-authtool and ceph-auth commands specified in radosgw(8)? > > Not for setting up radosgw. > > >> log file = /var/log/radosgw/radosgw.log > >> debug rgw = 20 > >> rgw cache enabled = 1 > >> ; rgw swift url = http://skinny > >> ; rgw swift url prefix = swift > > > > I ran across this option sifting through src/rgw, can you explain what > > the URL prefix is used for? > > When authenticating the client, the swift_url and swift_prefix are > concatenated and passed to the client as the storage URL, along with > the token. > > Thanks, > Yehuda > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo <at> vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: "Radosgw installation and administration" docs 2012-07-01 20:22 ` Chuanyu @ 2012-07-02 7:06 ` Florian Haas 2012-07-02 9:35 ` Chuanyu Tsai 2012-07-03 17:45 ` "Radosgw installation and administration" docs Yehuda Sadeh 2 siblings, 0 replies; 9+ messages in thread From: Florian Haas @ 2012-07-02 7:06 UTC (permalink / raw) To: Chuanyu; +Cc: ceph-devel On Sun, Jul 1, 2012 at 10:22 PM, Chuanyu <chuanyu@cs.nctu.edu.tw> wrote: > Hi Yehuda, Florian, > > I follow the wiki, and steps which you discussed, > construct my ceph system with rados gateway, > and I can use libs3 to upload file via radosgw, (thanks a lot!) > but got "405 Method Not Allowed" when I use swift, > > $ swift -v -A http://s3.paca.tw:80/auth -U paca:paca1 -K > UoJO4nFgdAoX+9nEftElIY+AMmDIkcrUBkycNKPA stat > Auth GET failed: http://s3.paca.tw:80/auth/tokens 405 Method Not Allowed > > ( Because there has no test step on wiki, > I follow the Florian's question, and guess the test command is above ?!) > > my radosgw-admin config: > $ radosgw-admin user info --uid=paca > { "user_id": "paca", > "rados_uid": 0, > "display_name": "chuanyu", > "email": "chuanyu@cs.nctu.edu.tw", > "suspended": 0, > "subusers": [ > { "id": "paca:paca1", > "permissions": "<none>"}], This is most likely your problem. You're being bitten by http://tracker.newdream.net/issues/2650. Try "radosgw-admin subuser modify --subuser=paca:paca1 --access=full" and see if that improves things. Cheers, Florian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2012-07-01 20:22 ` Chuanyu 2012-07-02 7:06 ` Florian Haas @ 2012-07-02 9:35 ` Chuanyu Tsai 2012-07-03 17:45 ` "Radosgw installation and administration" docs Yehuda Sadeh 2 siblings, 0 replies; 9+ messages in thread From: Chuanyu Tsai @ 2012-07-02 9:35 UTC (permalink / raw) To: ceph-devel Chuanyu <chuanyu <at> cs.nctu.edu.tw> writes: > Hi Yehuda, Florian, > > I follow the wiki, and steps which you discussed, > construct my ceph system with rados gateway, > and I can use libs3 to upload file via radosgw, (thanks a lot!) > but got "405 Method Not Allowed" when I use swift, > > $ swift -v -A http://s3.paca.tw:80/auth -U paca:paca1 -K > UoJO4nFgdAoX+9nEftElIY+AMmDIkcrUBkycNKPA stat > Auth GET failed: http://s3.paca.tw:80/auth/tokens 405 Method Not Allowed > > ( Because there has no test step on wiki, > I follow the Florian's question, and guess the test command is above ?!) > > my radosgw-admin config: > $ radosgw-admin user info --uid=paca > { "user_id": "paca", > "rados_uid": 0, > "display_name": "chuanyu", > "email": "chuanyu <at> cs.nctu.edu.tw", > "suspended": 0, > "subusers": [ > { "id": "paca:paca1", > "permissions": "full-control"}], I've correct the permissions problem, thanks Florian! > "keys": [ > { "user": "paca", > "access_key": "DS932H4EI9HK7I1CTDNF", > "secret_key": "Rn\/5FqHzRPZFN6f9R\/LuTqvG0AYjbHtrurrGydVk"}], > "swift_keys": [ > { "user": "paca:paca1", > "secret_key": "UoJO4nFgdAoX+9nEftElIY+AMmDIkcrUBkycNKPA"}]} > > ceph.conf: > [client.radosgw.gateway] > host = volume > keyring = /etc/ceph/keyring/radosgw.gateway.keyring > rgw socket path = /var/run/ceph/rgw.sock > log file = "" > syslog = true > debug rgw = 20 > > my log: > http://pastebin.com/rhGhATmv Hi, I've noticed that the log shows I'm using *POST* method to getting op? req 9:0.000277:swift-auth:POST /auth/tokens::getting op But the code shows I'll always get NULL return /ceph/src/rgw/rgw_swift_auth.cc:239 239 RGWOp *RGWHandler_SWIFT_Auth::get_op() 240 { 241 RGWOp *op; 242 switch (s->op) { 243 case OP_GET: 244 op = &rgw_swift_auth_get; 245 break; 246 default: 247 return NULL; 248 } So 405 error occurs, /ceph/src/rgw/rgw_main.cc:273 273 req->log(s, "getting op"); 274 op = handler->get_op(); 275 if (!op) { 276 abort_early(s, -ERR_METHOD_NOT_ALLOWED); 277 goto done; My swift version (Version: 1.4.8-0ubuntu2, Ubuntu 12.04) $ swift --version swift 1.0 Does the version mismatch, or something else goes wrong? I'll try curl connection directly later, Thanks! Chuanyu Tsai. > > Any advice would be appreciate! > Tthanks, > Chuanyu ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: "Radosgw installation and administration" docs 2012-07-01 20:22 ` Chuanyu 2012-07-02 7:06 ` Florian Haas 2012-07-02 9:35 ` Chuanyu Tsai @ 2012-07-03 17:45 ` Yehuda Sadeh 2 siblings, 0 replies; 9+ messages in thread From: Yehuda Sadeh @ 2012-07-03 17:45 UTC (permalink / raw) To: Chuanyu; +Cc: ceph-devel On Sun, Jul 1, 2012 at 1:22 PM, Chuanyu <chuanyu@cs.nctu.edu.tw> wrote: > Yehuda Sadeh <yehuda <at> inktank.com> writes: > >> >> On Tue, Jun 12, 2012 at 11:11 AM, Florian Haas <florian <at> hastexo.com> > wrote: >> > Hi Yehuda, >> > >> > thanks, that resolved a lot of questions for me. A few follow-up >> > comments below: >> > >> >> >> >> We currently use a slightly different rule: >> >> >> >> RewriteRule ^/(.*) >> >> /radosgw.fcgi?params=$1&%{QUERY_STRING} >> >> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] >> > >> > Could you explain what happened to "page"? >> >> Not really. I don't remember, was probably necessary originally and >> now it's not. Looking at the code, I think you can also drop the >> params=$1 part: >> >> RewriteRule ^/(.*) /radosgw.fcgi?%{QUERY_STRING} >> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] >> >> > >> >>> Also, for each of these, where would the logging output end up? >> >>> /var/log/ceph? Apache error log? If so, only if the Apache LogLevel is >> >>> more verbose than info? Syslog? >> >> >> >> >> >> The debug log would end up wherever you specified in the 'log file' >> >> config option. >> > >> > ... or syslog, if log file = "" and syslog = true. (iirc) >> >> Yeah. Whichever ceph logging scheme you're using. >> >> > >> >>> 6. Swift API: Keys >> >>> >> >>> Is it correct to assume that for any Swift client to work, we must set a >> >>> Swift key for the user, like so? >> >>> >> >>> radosgw-admin key create --key-type=swift --uid=<user> >> >>> >> >>> If so, is the secret_key that that creates for the user: >> >>> >> >>> "swift_keys": [ >> >>> { "user": "<user>", >> >>> "secret_key": "<longbase64hash>"}]} >> >>> >> >>> >> >>> ... the same key that the swift command line client expects to be set >> >>> with th -K option? >> >> >> >> Yes. >> > >> > OK, but I realized that you apparently have to create a separate key >> > when creating a sub-user. Is that correct? Or is there a way for >> > sub-users to "inherit" the keys defined for their parent user? >> > >> >>> 7. Swift API: swift user name >> >>> >> >>> When we call "swift -U <user>", is that the verbatim user_id that we've >> >>> defined with "radosgw-admin user create --uid=<user_id>"? Or do we need >> >>> to set a prefix? Or define a separate Swift user ID? >> >>> >> >> >> >> In swift the terminology is a bit different. There is the account and >> >> under that there is the user. Since we already have a 'user' (which is >> >> actually the swift account), we created a 'subuser'. So a one liner >> >> user and swift-subuser creation would be as follows: >> >> >> >> # radosgw-admin user create --subuser=yehuda:yehuda1 >> >> --display-name=Yehuda --key-type=swift --access=full >> > >> > It seems there is some magic involved so that if you do "user create", >> > set --subuser=<prefix>:<sub> and don't set --uid, it creates a new >> > parent user named <prefix>. Is this meant to be stable? Or is the >> >> The <user>:<subuser> notation is stable. >> >> > supported way of doing things to always first create a user, and then >> > use "subuser create" to create the subuser? >> >> Both are supported, but note that the 'user create' command requires a >> display-name to be specified, whereas the 'subuser create' doesn't. We >> can change that later and only require the display-name if the user >> does not exist, but at the moment that's how it works. >> >> > >> >> { "user_id": "yehuda", >> >> "rados_uid": 0, >> >> "display_name": "Yehuda", >> >> "email": "", >> >> "suspended": 0, >> >> "max_buckets": 1000, >> >> "subusers": [ >> >> { "id": "yehuda:yehuda1", >> >> "permissions": "full-control"}], >> >> "keys": [], >> >> "swift_keys": [ >> >> { "user": "yehuda:yehuda1", >> >> "secret_key": "7TD5f2QrwxkCnhthwowC4d9uEJ4mnX8nGsXjmnW8"}]} >> >> >> > > Hi Yehuda, Florian, > > I follow the wiki, and steps which you discussed, > construct my ceph system with rados gateway, > and I can use libs3 to upload file via radosgw, (thanks a lot!) > but got "405 Method Not Allowed" when I use swift, > > $ swift -v -A http://s3.paca.tw:80/auth -U paca:paca1 -K > UoJO4nFgdAoX+9nEftElIY+AMmDIkcrUBkycNKPA stat > Auth GET failed: http://s3.paca.tw:80/auth/tokens 405 Method Not Allowed > > ( Because there has no test step on wiki, > I follow the Florian's question, and guess the test command is above ?!) > > my radosgw-admin config: > $ radosgw-admin user info --uid=paca > { "user_id": "paca", > "rados_uid": 0, > "display_name": "chuanyu", > "email": "chuanyu@cs.nctu.edu.tw", > "suspended": 0, > "subusers": [ > { "id": "paca:paca1", > "permissions": "<none>"}], > "keys": [ > { "user": "paca", > "access_key": "DS932H4EI9HK7I1CTDNF", > "secret_key": "Rn\/5FqHzRPZFN6f9R\/LuTqvG0AYjbHtrurrGydVk"}], > "swift_keys": [ > { "user": "paca:paca1", > "secret_key": "UoJO4nFgdAoX+9nEftElIY+AMmDIkcrUBkycNKPA"}]} > > ceph.conf: > [client.radosgw.gateway] > host = volume > keyring = /etc/ceph/keyring/radosgw.gateway.keyring > rgw socket path = /var/run/ceph/rgw.sock > log file = "" > syslog = true > debug rgw = 20 > > my log: > http://pastebin.com/rhGhATmv > For some reason the swift tool here tries to do a POST on /auth/tokens. In any case, we don't support that, that's why you're getting 405. Try to stat a container/object and see if that works for you. In any case, Florian's answer is correct and you're also getting bit by issue #2650. Yehuda ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2012-07-03 17:45 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <4FD71854.6060503@hastexo.com> 2012-06-12 10:44 ` "Radosgw installation and administration" docs Florian Haas 2012-06-12 16:47 ` Yehuda Sadeh 2012-06-12 18:11 ` Florian Haas 2012-06-12 18:54 ` Yehuda Sadeh 2012-06-28 7:50 ` Florian Haas 2012-07-01 20:22 ` Chuanyu 2012-07-02 7:06 ` Florian Haas 2012-07-02 9:35 ` Chuanyu Tsai 2012-07-03 17:45 ` "Radosgw installation and administration" docs Yehuda Sadeh
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.